[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:57:44 dist-test-slave-30t4 krb5kdc[7361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:57:44 dist-test-slave-30t4 krb5kdc[7361](info): set up 2 sockets
Jun 20 14:57:44 dist-test-slave-30t4 krb5kdc[7361](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:57:45.484196 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:57:45 dist-test-slave-30t4 krb5kdc[7361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:57:45.598374 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:41607
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:41607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:45.703131 7374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:45.703367 7374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:45.703405 7374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:45.707227 7374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:45.707273 7374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:45.707291 7374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:45.707307 7374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:45.707321 7374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:45.712270 7374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:41607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:41607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:45.713136 7374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:45.714026 7374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:57:45.722497 7374 server_base.cc:1034] running on GCE node
W20260620 14:57:45.722473 7380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:45.722515 7379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:45.722693 7382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:45.723274 7374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:45.724354 7374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:45.725505 7374 hybrid_clock.cc:648] HybridClock initialized: now 1781967465725477 us; error 43 us; skew 500 ppm
Jun 20 14:57:45 dist-test-slave-30t4 krb5kdc[7361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967465, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:45.728029 7374 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:57:45.728868 7374 webserver.cc:460] Webserver started at http://127.7.46.190:36523/ using document root <none> and password file <none>
I20260620 14:57:45.729398 7374 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:45.729483 7374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:45.729705 7374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:45.731594 7374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "0de41dea035d43178c19210cd2c007f7"
format_stamp: "Formatted at 2026-06-20 14:57:45 on dist-test-slave-30t4"
server_key: "5ed8ad8832a22109a8a46743e7d86903"
server_key_iv: "c64519f03ba28056ac43af3092159cbf"
server_key_version: "encryptionkey@0"
I20260620 14:57:45.732203 7374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "0de41dea035d43178c19210cd2c007f7"
format_stamp: "Formatted at 2026-06-20 14:57:45 on dist-test-slave-30t4"
server_key: "5ed8ad8832a22109a8a46743e7d86903"
server_key_iv: "c64519f03ba28056ac43af3092159cbf"
server_key_version: "encryptionkey@0"
I20260620 14:57:45.735862 7374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:57:45.738130 7389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:45.739317 7374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:45.739454 7374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "0de41dea035d43178c19210cd2c007f7"
format_stamp: "Formatted at 2026-06-20 14:57:45 on dist-test-slave-30t4"
server_key: "5ed8ad8832a22109a8a46743e7d86903"
server_key_iv: "c64519f03ba28056ac43af3092159cbf"
server_key_version: "encryptionkey@0"
I20260620 14:57:45.739562 7374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:45.772421 7374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:45.775358 7374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:45.775556 7374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:45.782472 7374 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:41607
I20260620 14:57:45.782498 7441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:41607 every 8 connection(s)
I20260620 14:57:45.783380 7374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:45.784190 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 7374
I20260620 14:57:45.784339 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:57:45.785580 7354 external_mini_cluster.cc:1428] Setting key 74f287a218880b23828e4d69cdf24329
I20260620 14:57:45.786195 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:45.792153 7442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7: Bootstrap starting.
I20260620 14:57:45.794236 7442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:45.794952 7442 log.cc:824] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:45.796731 7442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7: No bootstrap required, opened a new log
I20260620 14:57:45.800186 7442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de41dea035d43178c19210cd2c007f7" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41607 } }
I20260620 14:57:45.800413 7442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:45.800498 7442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0de41dea035d43178c19210cd2c007f7, State: Initialized, Role: FOLLOWER
I20260620 14:57:45.800959 7442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de41dea035d43178c19210cd2c007f7" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41607 } }
I20260620 14:57:45.801106 7442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:45.801194 7442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:45.801288 7442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:57:45 dist-test-slave-30t4 krb5kdc[7361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:45.802153 7442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de41dea035d43178c19210cd2c007f7" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41607 } }
I20260620 14:57:45.802474 7442 leader_election.cc:302] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0de41dea035d43178c19210cd2c007f7; no voters:
I20260620 14:57:45.802815 7442 leader_election.cc:288] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:57:45.802924 7447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:45.803134 7447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [term 1 LEADER]: Becoming Leader. State: Replica: 0de41dea035d43178c19210cd2c007f7, State: Running, Role: LEADER
I20260620 14:57:45.803457 7447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de41dea035d43178c19210cd2c007f7" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41607 } }
I20260620 14:57:45.804064 7442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:45.805109 7448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0de41dea035d43178c19210cd2c007f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de41dea035d43178c19210cd2c007f7" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41607 } } }
I20260620 14:57:45.805225 7448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:45.805521 7449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0de41dea035d43178c19210cd2c007f7. Latest consensus state: current_term: 1 leader_uuid: "0de41dea035d43178c19210cd2c007f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de41dea035d43178c19210cd2c007f7" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41607 } } }
I20260620 14:57:45.805632 7449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:45.805873 7456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:45.808898 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:45.788552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.7.46.190:41607)
0620 14:57:45.788942 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:45.788958 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:45.789990 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 14:57:45.792919 (+ 2929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:45.792933 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:45.793214 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:45.793536 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:45.795683 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:45.796701 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:45.797843 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:45.798282 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:45.798447 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:45.802340 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:45.802346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:45.802367 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:45.802392 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:45.804473 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:45.805175 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:45.805181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:45.805186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:45.805438 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:45.806029 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:45.806034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:45.806038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:45.806555 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:45.806761 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:45.807405 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 14:57:45.807676 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20260620 14:57:45.809271 7456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:45.815073 7456 catalog_manager.cc:1347] Generated new cluster ID: cf1bdfa1f97148aeab150fb24300b5d8
I20260620 14:57:45.815143 7456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:57:45.822146 7456 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:57:45.822937 7456 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:57:45.829911 7456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0de41dea035d43178c19210cd2c007f7: Generated new TSK 0
I20260620 14:57:45.830540 7456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:57:45.856715 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41607
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:45.964315 7468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:45.964615 7468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:45.964691 7468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:45.968684 7468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:45.968758 7468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:45.968845 7468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:57:45.974103 7468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:45.975121 7468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:45.976125 7468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:45.985339 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:45.985339 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:45.985491 7468 server_base.cc:1034] running on GCE node
W20260620 14:57:45.985581 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:45.986043 7468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:45.986636 7468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:45.987836 7468 hybrid_clock.cc:648] HybridClock initialized: now 1781967465987803 us; error 44 us; skew 500 ppm
Jun 20 14:57:45 dist-test-slave-30t4 krb5kdc[7361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967465, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:45.990475 7468 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:57:45.991307 7468 webserver.cc:460] Webserver started at http://127.7.46.129:42251/ using document root <none> and password file <none>
I20260620 14:57:45.991902 7468 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:45.991989 7468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:45.992216 7468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:45.994160 7468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "ffc9cd68e32144568bf97890f48e56c1"
format_stamp: "Formatted at 2026-06-20 14:57:45 on dist-test-slave-30t4"
server_key: "0824ef463c9d7e58b374c80402b196b8"
server_key_iv: "d47ef188d1338ba85fcb1fd1990f4fa7"
server_key_version: "encryptionkey@0"
I20260620 14:57:45.994717 7468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "ffc9cd68e32144568bf97890f48e56c1"
format_stamp: "Formatted at 2026-06-20 14:57:45 on dist-test-slave-30t4"
server_key: "0824ef463c9d7e58b374c80402b196b8"
server_key_iv: "d47ef188d1338ba85fcb1fd1990f4fa7"
server_key_version: "encryptionkey@0"
I20260620 14:57:45.998414 7468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:57:46.000764 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.001897 7468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:57:46.002038 7468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "ffc9cd68e32144568bf97890f48e56c1"
format_stamp: "Formatted at 2026-06-20 14:57:45 on dist-test-slave-30t4"
server_key: "0824ef463c9d7e58b374c80402b196b8"
server_key_iv: "d47ef188d1338ba85fcb1fd1990f4fa7"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.002156 7468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:46.020869 7468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:46.023633 7468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:46.023883 7468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:46.024456 7468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:46.025255 7468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:46.025326 7468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.025398 7468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:46.025451 7468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.034211 7468 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:36469
I20260620 14:57:46.034238 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:36469 every 8 connection(s)
I20260620 14:57:46.035102 7468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:46.042656 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 7468
I20260620 14:57:46.042768 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:57:46.043093 7354 external_mini_cluster.cc:1428] Setting key 220ec56c16b75472995ee22e289bbc92
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967465, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:46.049198 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.036770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:38799 (local address 127.7.46.190:41607)
0620 14:57:46.036899 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:46.036902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:46.037681 (+ 779us) server_negotiation.cc:407] Connection header received
0620 14:57:46.039046 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:46.039050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:46.039106 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:46.039218 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:46.040388 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.040922 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.042037 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.042286 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.042399 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.045352 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:46.045358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:46.045364 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:46.045397 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:46.046949 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.047562 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.047568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.047573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.047663 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.048020 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.048026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.048031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.048291 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:46.048395 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:46.048921 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 14:57:46.049035 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:57:46.050314 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.037119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:41607 (local address 127.7.46.129:38799)
0620 14:57:46.037530 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:46.037571 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:46.038837 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:46.039385 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:46.039391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:46.039730 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:46.040242 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:46.040255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.041049 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.041054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:46.041906 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:46.041915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.042442 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.042446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:46.042724 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.043378 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:46.043393 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:46.045189 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:46.047076 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:46.047082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:46.047099 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:46.047457 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:46.047753 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:46.047755 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:46.047756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:46.047923 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:46.048400 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:46.048406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:46.048669 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 14:57:46.049347 (+ 678us) client_negotiation.cc:240] Negotiation successful
0620 14:57:46.049610 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20260620 14:57:46.051481 7597 heartbeater.cc:347] Connected to a master server at 127.7.46.190:41607
I20260620 14:57:46.051717 7597 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:46.052186 7597 heartbeater.cc:511] Master 127.7.46.190:41607 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 14:57:46.053619 7406 ts_manager.cc:194] Registered new tserver with Master: ffc9cd68e32144568bf97890f48e56c1 (127.7.46.129:36469)
I20260620 14:57:46.054883 7406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:38799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:57:46.068929 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41607
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:46.183028 7602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:46.183279 7602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:46.183354 7602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:46.187304 7602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:46.187386 7602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:46.187472 7602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:57:46.192848 7602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:46.193847 7602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:46.194836 7602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:46.204270 7608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:46.204274 7607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:46.204422 7602 server_base.cc:1034] running on GCE node
W20260620 14:57:46.204288 7610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:46.205302 7602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:46.205972 7602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:46.207171 7602 hybrid_clock.cc:648] HybridClock initialized: now 1781967466207133 us; error 49 us; skew 500 ppm
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:46.209827 7602 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:57:46.210705 7602 webserver.cc:460] Webserver started at http://127.7.46.130:38093/ using document root <none> and password file <none>
I20260620 14:57:46.211234 7602 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:46.211292 7602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:46.211485 7602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:46.213469 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "75aa555a9ac2473f933ea8cb8c60bb5e"
format_stamp: "Formatted at 2026-06-20 14:57:46 on dist-test-slave-30t4"
server_key: "ade507b11cbaa93ed06e972163883e12"
server_key_iv: "05db4c9dfe054934bc11061ecee061f2"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.214015 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "75aa555a9ac2473f933ea8cb8c60bb5e"
format_stamp: "Formatted at 2026-06-20 14:57:46 on dist-test-slave-30t4"
server_key: "ade507b11cbaa93ed06e972163883e12"
server_key_iv: "05db4c9dfe054934bc11061ecee061f2"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.217723 7602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:57:46.220175 7617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.221426 7602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:57:46.221560 7602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "75aa555a9ac2473f933ea8cb8c60bb5e"
format_stamp: "Formatted at 2026-06-20 14:57:46 on dist-test-slave-30t4"
server_key: "ade507b11cbaa93ed06e972163883e12"
server_key_iv: "05db4c9dfe054934bc11061ecee061f2"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.221673 7602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:46.249568 7602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:46.252745 7602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:46.252939 7602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:46.253542 7602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:46.254387 7602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:46.254451 7602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.254508 7602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:46.254544 7602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.265305 7602 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:39347
I20260620 14:57:46.265434 7730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:39347 every 8 connection(s)
I20260620 14:57:46.266202 7602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:57:46.274696 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 7602
I20260620 14:57:46.274808 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:57:46.275103 7354 external_mini_cluster.cc:1428] Setting key 87cf2d9b36908314fa44bd0b49a21438
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:46.280488 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.267985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:48229 (local address 127.7.46.190:41607)
0620 14:57:46.268148 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:46.268153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:46.268954 (+ 801us) server_negotiation.cc:407] Connection header received
0620 14:57:46.270283 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:46.270289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:46.270361 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:46.270487 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:46.271552 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.272637 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.273724 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.274075 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.274194 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.276700 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:46.276703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:46.276705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:46.276726 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:46.278163 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.278767 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.278769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.278770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.278838 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.279218 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.279220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.279240 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.279482 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:46.279585 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:46.280114 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 14:57:46.280249 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:57:46.281121 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.268358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:41607 (local address 127.7.46.130:48229)
0620 14:57:46.268788 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:46.268838 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:46.270063 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:46.270618 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:46.270624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:46.270947 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:46.271405 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:46.271418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.272771 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.272774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:46.273607 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:46.273615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.274225 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.274227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:46.274356 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.274954 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:46.274980 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:46.276546 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:46.278273 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:46.278279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:46.278294 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:46.278668 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:46.278928 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:46.278933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:46.278937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:46.279125 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:46.279592 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:46.279597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:46.279883 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 14:57:46.280042 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 14:57:46.280277 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
I20260620 14:57:46.282271 7731 heartbeater.cc:347] Connected to a master server at 127.7.46.190:41607
I20260620 14:57:46.282501 7731 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:46.282951 7731 heartbeater.cc:511] Master 127.7.46.190:41607 requested a full tablet report, sending...
I20260620 14:57:46.284001 7406 ts_manager.cc:194] Registered new tserver with Master: 75aa555a9ac2473f933ea8cb8c60bb5e (127.7.46.130:39347)
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
I20260620 14:57:46.284842 7406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:48229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:57:46.300793 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41607
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:46.411904 7736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:46.412153 7736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:46.412227 7736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:46.416275 7736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:46.416327 7736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:46.416419 7736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:57:46.421614 7736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:46.422623 7736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:46.423624 7736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:46.433056 7741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:46.432982 7742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:46.433207 7736 server_base.cc:1034] running on GCE node
W20260620 14:57:46.433001 7744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:46.433655 7736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:46.434286 7736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:46.435470 7736 hybrid_clock.cc:648] HybridClock initialized: now 1781967466435457 us; error 24 us; skew 500 ppm
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:46.438067 7736 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:57:46.438870 7736 webserver.cc:460] Webserver started at http://127.7.46.131:46323/ using document root <none> and password file <none>
I20260620 14:57:46.439406 7736 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:46.439502 7736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:46.439721 7736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:46.441701 7736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "76e6f3bab4ed4b189752f94e1c5eda41"
format_stamp: "Formatted at 2026-06-20 14:57:46 on dist-test-slave-30t4"
server_key: "c10bf32bc168b88a0a0c473b1f9f49ae"
server_key_iv: "113697d06a2ab754749d4ac5e541406b"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.442255 7736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "76e6f3bab4ed4b189752f94e1c5eda41"
format_stamp: "Formatted at 2026-06-20 14:57:46 on dist-test-slave-30t4"
server_key: "c10bf32bc168b88a0a0c473b1f9f49ae"
server_key_iv: "113697d06a2ab754749d4ac5e541406b"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.445912 7736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 14:57:46.448161 7751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.449306 7736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:57:46.449432 7736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "76e6f3bab4ed4b189752f94e1c5eda41"
format_stamp: "Formatted at 2026-06-20 14:57:46 on dist-test-slave-30t4"
server_key: "c10bf32bc168b88a0a0c473b1f9f49ae"
server_key_iv: "113697d06a2ab754749d4ac5e541406b"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.449539 7736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:46.470424 7736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:46.473220 7736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:46.473421 7736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:46.473981 7736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:46.474771 7736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:46.474839 7736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.474905 7736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:46.474954 7736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.483590 7736 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:33831
I20260620 14:57:46.483610 7864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:33831 every 8 connection(s)
I20260620 14:57:46.484490 7736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:57:46.486387 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 7736
I20260620 14:57:46.486515 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:57:46.486842 7354 external_mini_cluster.cc:1428] Setting key eb21d901eb4292a020266d1135b56384
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:46.497556 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.486375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:33175 (local address 127.7.46.190:41607)
0620 14:57:46.486538 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:46.486542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:46.487554 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 14:57:46.489071 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:46.489074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:46.489118 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:46.489197 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:46.490296 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.490824 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.491980 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.492191 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.492273 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.494546 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:46.494549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:46.494551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:46.494574 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:46.495836 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.496282 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.496284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.496286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.496356 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.496660 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.496663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.496664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.496832 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:46.496909 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:46.497341 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 14:57:46.497435 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:57:46.498322 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.486821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:41607 (local address 127.7.46.131:33175)
0620 14:57:46.487366 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:46.487417 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:46.488858 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:46.489344 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:46.489351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:46.489699 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:46.490172 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:46.490185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.490930 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.490932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:46.491878 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:46.491888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.492317 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.492320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:46.492452 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.492992 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:46.493008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:46.494401 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:46.495936 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:46.495941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:46.495952 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:46.496199 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:46.496432 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:46.496434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:46.496436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:46.496559 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:46.496920 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:46.496925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:46.497144 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 14:57:46.497297 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 14:57:46.497532 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":161,"threads_started":1}
I20260620 14:57:46.499397 7865 heartbeater.cc:347] Connected to a master server at 127.7.46.190:41607
I20260620 14:57:46.499634 7865 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:46.500166 7865 heartbeater.cc:511] Master 127.7.46.190:41607 requested a full tablet report, sending...
I20260620 14:57:46.501340 7405 ts_manager.cc:194] Registered new tserver with Master: 76e6f3bab4ed4b189752f94e1c5eda41 (127.7.46.131:33831)
I20260620 14:57:46.501915 7405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:33175
I20260620 14:57:46.513279 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:57:46.538028 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.529337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.7.46.129:36469)
0620 14:57:46.529619 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:46.529625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:46.529659 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:57:46.529708 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:46.529712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:46.529946 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:46.530068 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:46.530667 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.531412 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.532333 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.532695 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.532779 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.535075 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:46.535083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:46.535096 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:46.535116 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:46.536598 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.536976 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.536982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.536983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.537056 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.537273 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.537275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.537277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.537491 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:46.537647 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:46.537779 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:57:46.537893 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
W20260620 14:57:46.538944 7511 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43466
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:57:46.563345 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.555431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.7.46.129:36469)
0620 14:57:46.555573 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:46.555578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:46.555692 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:57:46.555873 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:46.555876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:46.555926 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:46.556010 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:46.556664 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.557170 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.557972 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.558207 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.558315 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.560573 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:46.560576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:46.560579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:46.560603 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:46.561784 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.562193 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.562198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.562200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.562278 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.562540 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.562544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.562546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.562702 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:46.562831 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:46.563067 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 14:57:46.563182 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:57:46.564457 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 7468
I20260620 14:57:46.570195 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 7602
I20260620 14:57:46.575244 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 7736
I20260620 14:57:46.580188 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 7374
2026-06-20T14:57:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2153 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7879](info): set up 2 sockets
Jun 20 14:57:46 dist-test-slave-30t4 krb5kdc[7879](info): commencing operation
krb5kdc: starting...
W20260620 14:57:47.623453 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:57:47 dist-test-slave-30t4 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:57:47.736932 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:33135
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:33135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:47.840339 7892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:47.840564 7892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:47.840601 7892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:47.844365 7892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:47.844408 7892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:47.844427 7892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:47.844444 7892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:47.844458 7892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:47.849282 7892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:33135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:47.850113 7892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:47.850972 7892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:47.858754 7897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:47.858770 7898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:47.858801 7900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:47.859071 7892 server_base.cc:1034] running on GCE node
I20260620 14:57:47.859691 7892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:47.860769 7892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:47.861922 7892 hybrid_clock.cc:648] HybridClock initialized: now 1781967467861903 us; error 29 us; skew 500 ppm
Jun 20 14:57:47 dist-test-slave-30t4 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967467, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:47.864833 7892 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:57:47.865818 7892 webserver.cc:460] Webserver started at http://127.7.46.190:37993/ using document root <none> and password file <none>
I20260620 14:57:47.866379 7892 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:47.866475 7892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:47.866719 7892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:47.868894 7892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "2414e673eb4141a0b0e7e5950d1c27ce"
format_stamp: "Formatted at 2026-06-20 14:57:47 on dist-test-slave-30t4"
server_key: "cdbc4c0b7e20bb0d005144a802d29c76"
server_key_iv: "48c81113a53d8a46cfba6edf9a69e923"
server_key_version: "encryptionkey@0"
I20260620 14:57:47.869481 7892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "2414e673eb4141a0b0e7e5950d1c27ce"
format_stamp: "Formatted at 2026-06-20 14:57:47 on dist-test-slave-30t4"
server_key: "cdbc4c0b7e20bb0d005144a802d29c76"
server_key_iv: "48c81113a53d8a46cfba6edf9a69e923"
server_key_version: "encryptionkey@0"
I20260620 14:57:47.873250 7892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:57:47.875574 7907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:47.876782 7892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:57:47.876921 7892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "2414e673eb4141a0b0e7e5950d1c27ce"
format_stamp: "Formatted at 2026-06-20 14:57:47 on dist-test-slave-30t4"
server_key: "cdbc4c0b7e20bb0d005144a802d29c76"
server_key_iv: "48c81113a53d8a46cfba6edf9a69e923"
server_key_version: "encryptionkey@0"
I20260620 14:57:47.877025 7892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:47.893096 7892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:47.895617 7892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:47.895841 7892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:47.902575 7892 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:33135
I20260620 14:57:47.902594 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:33135 every 8 connection(s)
I20260620 14:57:47.903461 7892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:47.906031 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:47.911120 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce: Bootstrap starting.
I20260620 14:57:47.912060 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 7892
I20260620 14:57:47.912195 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:57:47.912462 7354 external_mini_cluster.cc:1428] Setting key e7966621540a91272a7b6e8228f8b65c
I20260620 14:57:47.913822 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:47.914695 7960 log.cc:824] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:47.916760 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce: No bootstrap required, opened a new log
I20260620 14:57:47.920770 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2414e673eb4141a0b0e7e5950d1c27ce" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33135 } }
I20260620 14:57:47.921047 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:47.921115 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2414e673eb4141a0b0e7e5950d1c27ce, State: Initialized, Role: FOLLOWER
Jun 20 14:57:47 dist-test-slave-30t4 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:47.921604 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2414e673eb4141a0b0e7e5950d1c27ce" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33135 } }
I20260620 14:57:47.921772 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:47.921842 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:47.921950 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:47.923054 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2414e673eb4141a0b0e7e5950d1c27ce" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33135 } }
I20260620 14:57:47.923394 7960 leader_election.cc:302] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2414e673eb4141a0b0e7e5950d1c27ce; no voters:
I20260620 14:57:47.923714 7960 leader_election.cc:288] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:57:47.923803 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:47.923985 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [term 1 LEADER]: Becoming Leader. State: Replica: 2414e673eb4141a0b0e7e5950d1c27ce, State: Running, Role: LEADER
I20260620 14:57:47.924275 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2414e673eb4141a0b0e7e5950d1c27ce" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33135 } }
I20260620 14:57:47.924970 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:47.925374 7965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2414e673eb4141a0b0e7e5950d1c27ce. Latest consensus state: current_term: 1 leader_uuid: "2414e673eb4141a0b0e7e5950d1c27ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2414e673eb4141a0b0e7e5950d1c27ce" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33135 } } }
I20260620 14:57:47.925478 7965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:47.925761 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2414e673eb4141a0b0e7e5950d1c27ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2414e673eb4141a0b0e7e5950d1c27ce" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33135 } } }
I20260620 14:57:47.925853 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:47.927703 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.913539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.7.46.190:33135)
0620 14:57:47.914002 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.914023 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.914083 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:57:47.914865 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.914887 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.915263 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.915603 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:47.916269 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.917660 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.918457 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.919156 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.919329 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.922028 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:47.922037 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:47.922055 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:47.922084 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:47.923949 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.924469 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.924474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.924480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.924630 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.924913 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.924916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.924918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.925412 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:47.926205 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.926344 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.926655 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":116,"threads_started":1}
W20260620 14:57:47.929771 7980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:57:47.929872 7980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:57:47.930119 7969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:47.930806 7969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:47.936388 7969 catalog_manager.cc:1347] Generated new cluster ID: 2520eab6281e45d5bed30b304e3f36d7
I20260620 14:57:47.936456 7969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:57:47.947202 7969 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:57:47.948210 7969 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:57:47.954743 7969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2414e673eb4141a0b0e7e5950d1c27ce: Generated new TSK 0
I20260620 14:57:47.955379 7969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:57:47.986770 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33135
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:48.112380 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:48.112624 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:48.112707 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:48.116781 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:48.116835 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:48.116942 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:57:48.122181 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:48.123221 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:48.124284 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:48.134045 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:48.134122 7986 server_base.cc:1034] running on GCE node
W20260620 14:57:48.134063 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:48.134058 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:48.134589 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:48.135196 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:48.136348 7986 hybrid_clock.cc:648] HybridClock initialized: now 1781967468136337 us; error 25 us; skew 500 ppm
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:48.138947 7986 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:57:48.139756 7986 webserver.cc:460] Webserver started at http://127.7.46.129:33407/ using document root <none> and password file <none>
I20260620 14:57:48.140362 7986 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:48.140420 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:48.140647 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:48.142604 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "7319253dfff34395ad709aa7562f74c5"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "8ba72a2f83176936e8f21941af90a0bf"
server_key_iv: "7f423ddd8622f331bda4fd800eeae6b0"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.143185 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "7319253dfff34395ad709aa7562f74c5"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "8ba72a2f83176936e8f21941af90a0bf"
server_key_iv: "7f423ddd8622f331bda4fd800eeae6b0"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.146708 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 14:57:48.149019 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.150154 7986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:57:48.150277 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "7319253dfff34395ad709aa7562f74c5"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "8ba72a2f83176936e8f21941af90a0bf"
server_key_iv: "7f423ddd8622f331bda4fd800eeae6b0"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.150378 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:48.180843 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:48.183653 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:48.183877 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:48.184451 7986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:48.185247 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:48.185333 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.185408 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:48.185462 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.194317 7986 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:40925
I20260620 14:57:48.194339 8114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:40925 every 8 connection(s)
I20260620 14:57:48.195179 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:48.203262 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 7986
I20260620 14:57:48.203413 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:57:48.203683 7354 external_mini_cluster.cc:1428] Setting key a18d0005a93d431cc2d8336b85ba8a95
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:48.209103 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.196927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:45245 (local address 127.7.46.190:33135)
0620 14:57:48.197057 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.197061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.197844 (+ 783us) server_negotiation.cc:407] Connection header received
0620 14:57:48.199165 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.199169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.199223 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.199302 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.200496 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.201028 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.202213 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.202463 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.202576 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.205199 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.205202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.205205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.205232 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.206908 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.207521 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.207524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.207525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.207599 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.208026 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.208029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.208030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.208195 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.208270 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.208882 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.208983 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:57:48.210227 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.197281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33135 (local address 127.7.46.129:45245)
0620 14:57:48.197701 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:48.197741 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:48.198960 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:48.199462 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:48.199469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:48.199879 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:48.200350 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.200363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.201172 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.201175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.202079 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.202089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.202613 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.202617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.202786 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.203376 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:48.203417 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:48.205026 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:48.207050 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:48.207058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:48.207074 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:48.207426 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:48.207701 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:48.207706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:48.207709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:48.207919 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:48.208300 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:48.208309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:48.208635 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 14:57:48.208830 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 14:57:48.209167 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
I20260620 14:57:48.211475 8115 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33135
I20260620 14:57:48.211716 8115 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:48.212229 8115 heartbeater.cc:511] Master 127.7.46.190:33135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 14:57:48.213671 7924 ts_manager.cc:194] Registered new tserver with Master: 7319253dfff34395ad709aa7562f74c5 (127.7.46.129:40925)
I20260620 14:57:48.214900 7924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:45245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:57:48.231062 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33135
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:48.334991 8120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:48.335247 8120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:48.335332 8120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:48.339239 8120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:48.339316 8120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:48.339397 8120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:57:48.344435 8120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:48.345409 8120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:48.346329 8120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:48.355185 8126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:48.355185 8125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:48.355191 8128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:48.355733 8120 server_base.cc:1034] running on GCE node
I20260620 14:57:48.356168 8120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:48.356724 8120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:48.357899 8120 hybrid_clock.cc:648] HybridClock initialized: now 1781967468357892 us; error 29 us; skew 500 ppm
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:48.360580 8120 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:57:48.361393 8120 webserver.cc:460] Webserver started at http://127.7.46.130:37289/ using document root <none> and password file <none>
I20260620 14:57:48.361905 8120 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:48.361994 8120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:48.362217 8120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:48.364163 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "6dcc64e574f74303a0f4bd483143b51b"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "08761adcb639f79dc5a652d90464debd"
server_key_iv: "f7c077a2a7d3cb9638989621b3cda359"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.364689 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "6dcc64e574f74303a0f4bd483143b51b"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "08761adcb639f79dc5a652d90464debd"
server_key_iv: "f7c077a2a7d3cb9638989621b3cda359"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.368073 8120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:57:48.370246 8135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.371356 8120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:48.371482 8120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "6dcc64e574f74303a0f4bd483143b51b"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "08761adcb639f79dc5a652d90464debd"
server_key_iv: "f7c077a2a7d3cb9638989621b3cda359"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.371598 8120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:48.402160 8120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:48.404971 8120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:48.405174 8120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:48.405740 8120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:48.406527 8120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:48.406596 8120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.406661 8120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:48.406705 8120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.415566 8120 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:43323
I20260620 14:57:48.415609 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:43323 every 8 connection(s)
I20260620 14:57:48.416468 8120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:48.426558 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 8120
I20260620 14:57:48.426697 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:57:48.427029 7354 external_mini_cluster.cc:1428] Setting key 225c30f69c13ddb7ef8c78f32e4ef497
I20260620 14:57:48.430176 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.418187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:48787 (local address 127.7.46.190:33135)
0620 14:57:48.418316 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.418320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.419101 (+ 781us) server_negotiation.cc:407] Connection header received
0620 14:57:48.420460 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.420464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.420518 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.420621 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.421730 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.422254 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.423351 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.423572 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.423682 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.426194 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.426197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.426199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.426222 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.427940 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.428533 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.428536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.428538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.428627 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.428939 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.428943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.428945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.429188 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.429297 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.429798 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.429934 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:57:48.431334 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.418547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33135 (local address 127.7.46.130:48787)
0620 14:57:48.418961 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:48.418999 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:48.420263 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:48.420809 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:48.420815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:48.421147 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:48.421583 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.421605 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.422378 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.422387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.423232 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.423240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.423715 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.423719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.423911 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.424453 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:48.424467 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:48.425941 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:48.428050 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:48.428056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:48.428070 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:48.428422 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:48.428722 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:48.428725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:48.428726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:48.428847 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:48.429316 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:48.429330 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:48.429560 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 14:57:48.429722 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:57:48.429965 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260620 14:57:48.432590 8249 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33135
I20260620 14:57:48.432869 8249 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:48.433341 8249 heartbeater.cc:511] Master 127.7.46.190:33135 requested a full tablet report, sending...
I20260620 14:57:48.434437 7924 ts_manager.cc:194] Registered new tserver with Master: 6dcc64e574f74303a0f4bd483143b51b (127.7.46.130:43323)
I20260620 14:57:48.435029 7924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:48787
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:57:48.453948 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33135
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:48.559682 8254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:48.559984 8254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:48.560066 8254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:48.563948 8254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:48.564024 8254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:48.564111 8254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:57:48.569206 8254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:48.570152 8254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:48.571102 8254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:48.580235 8259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:48.580240 8262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:48.580255 8260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:48.580544 8254 server_base.cc:1034] running on GCE node
I20260620 14:57:48.580955 8254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:48.581521 8254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:48.582711 8254 hybrid_clock.cc:648] HybridClock initialized: now 1781967468582701 us; error 30 us; skew 500 ppm
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:48.585316 8254 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:57:48.586187 8254 webserver.cc:460] Webserver started at http://127.7.46.131:46607/ using document root <none> and password file <none>
I20260620 14:57:48.586722 8254 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:48.586808 8254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:48.587023 8254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:48.589017 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "ada254838cf64ffb945369ab6233756c"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "ec78a28299228165c74c2fd0365330d8"
server_key_iv: "2fb73afbecc7569b8b3cb126852166c7"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.589561 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "ada254838cf64ffb945369ab6233756c"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "ec78a28299228165c74c2fd0365330d8"
server_key_iv: "2fb73afbecc7569b8b3cb126852166c7"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.593099 8254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:57:48.595327 8269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.596549 8254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:57:48.596692 8254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "ada254838cf64ffb945369ab6233756c"
format_stamp: "Formatted at 2026-06-20 14:57:48 on dist-test-slave-30t4"
server_key: "ec78a28299228165c74c2fd0365330d8"
server_key_iv: "2fb73afbecc7569b8b3cb126852166c7"
server_key_version: "encryptionkey@0"
I20260620 14:57:48.596797 8254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:48.610646 8254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:48.613319 8254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:48.613510 8254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:48.614097 8254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:48.614881 8254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:48.614951 8254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.615020 8254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:48.615063 8254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:48.623545 8254 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:39571
I20260620 14:57:48.623566 8382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:39571 every 8 connection(s)
I20260620 14:57:48.624392 8254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:57:48.629725 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 8254
I20260620 14:57:48.629861 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:57:48.630153 7354 external_mini_cluster.cc:1428] Setting key c65288a8b308ab4fed6605fa1c791af2
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967468, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:48.637363 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.626160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:50873 (local address 127.7.46.190:33135)
0620 14:57:48.626316 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.626320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.627099 (+ 779us) server_negotiation.cc:407] Connection header received
0620 14:57:48.628488 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.628491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.628548 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.628656 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.629870 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.630479 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.631760 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.632065 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.632188 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.634399 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.634402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.634404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.634427 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.635613 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.636122 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.636125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.636126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.636207 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.636489 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.636492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.636493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.636656 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.636732 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.637155 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.637250 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:57:48.638123 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.626531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33135 (local address 127.7.46.131:50873)
0620 14:57:48.626960 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:48.627000 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:48.628288 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:48.628814 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:48.628820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:48.629156 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:48.629703 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.629726 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.630604 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.630608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.631656 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.631665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.632220 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.632224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.632402 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.632905 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:48.632920 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:48.634253 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:48.635709 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:48.635713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:48.635724 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:48.636017 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:48.636294 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:48.636296 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:48.636297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:48.636414 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:48.636739 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:48.636743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:48.636946 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 14:57:48.637091 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 14:57:48.637318 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260620 14:57:48.639173 8383 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33135
I20260620 14:57:48.639405 8383 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:48.639925 8383 heartbeater.cc:511] Master 127.7.46.190:33135 requested a full tablet report, sending...
I20260620 14:57:48.640865 7924 ts_manager.cc:194] Registered new tserver with Master: ada254838cf64ffb945369ab6233756c (127.7.46.131:39571)
I20260620 14:57:48.641366 7924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:50873
I20260620 14:57:48.643831 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:57:48.653787 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.647018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.7.46.190:33135)
0620 14:57:48.647141 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.647145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.647354 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:57:48.647446 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.647448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.647483 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.647550 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.648382 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.648992 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.649739 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.649936 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.650023 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.651217 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.651220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.651221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.651243 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.652509 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.652868 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.652870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.652871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.652939 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.653150 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.653152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.653154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.653336 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.653433 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.653601 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.653681 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:57:48.658196 7924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:57:48.661252 7924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:48.675143 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.669339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.7.46.129:40925)
0620 14:57:48.670109 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.670128 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.670154 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:57:48.670232 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.670237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.670448 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.670592 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:48.670890 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.671984 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.673431 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.674198 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.674324 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.674457 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.674874 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.674968 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":108,"threads_started":1}
I20260620 14:57:48.675202 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.669959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.7.46.130:43323)
0620 14:57:48.670700 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.670714 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.670737 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:57:48.670811 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.670816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.670985 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.671138 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:48.671361 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.672446 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.673861 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.674601 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.674736 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.674863 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.674961 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.675051 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":140,"threads_started":1}
I20260620 14:57:48.675197 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.669576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:43323 (local address 127.0.0.1:38838)
0620 14:57:48.669823 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:48.669837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:48.669913 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:48.671116 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:48.671119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:48.671137 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:48.671240 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.671247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.672587 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.672590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.673742 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.673752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.674735 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.674739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.674824 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.674845 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:57:48.674933 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:57:48.675052 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":34,"thread_start_us":101,"threads_started":1}
I20260620 14:57:48.675575 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.669197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:40925 (local address 127.0.0.1:58866)
0620 14:57:48.669521 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:48.669569 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:48.669716 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:48.670610 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:48.670616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:48.670637 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:48.670759 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.670766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.672109 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.672113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.673293 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.673303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.674549 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.674553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.674666 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.674704 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:57:48.675391 (+ 687us) client_negotiation.cc:240] Negotiation successful
0620 14:57:48.675459 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260620 14:57:48.676052 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.669886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:39571 (local address 127.0.0.1:33542)
0620 14:57:48.670384 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:48.670396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:48.670476 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:48.671450 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:48.671453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:48.671470 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:48.671565 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.671571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.672874 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.672879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.673978 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.673988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.675642 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.675647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.675759 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.675802 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:57:48.675866 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:57:48.675929 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":20,"thread_start_us":80,"threads_started":1}
I20260620 14:57:48.676769 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.670255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.7.46.131:39571)
0620 14:57:48.670890 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.670902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.670921 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:57:48.670988 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.670993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.671175 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.671309 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:48.671687 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.672753 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.674107 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.674941 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.675032 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.675191 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.676527 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.676630 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":134,"threads_started":1}
I20260620 14:57:48.678210 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 70ec116a56174985934b6dd86d3e4a86 (DEFAULT_TABLE table=test-table [id=b11617301aec457b84850f4aada11deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:48.678264 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 70ec116a56174985934b6dd86d3e4a86 (DEFAULT_TABLE table=test-table [id=b11617301aec457b84850f4aada11deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:48.679147 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ec116a56174985934b6dd86d3e4a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:48.679245 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ec116a56174985934b6dd86d3e4a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:48.679075 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 70ec116a56174985934b6dd86d3e4a86 (DEFAULT_TABLE table=test-table [id=b11617301aec457b84850f4aada11deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:48.680032 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ec116a56174985934b6dd86d3e4a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:48.685034 8400 tablet_bootstrap.cc:492] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5: Bootstrap starting.
I20260620 14:57:48.686311 8401 tablet_bootstrap.cc:492] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b: Bootstrap starting.
I20260620 14:57:48.687431 8400 tablet_bootstrap.cc:654] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:48.688872 8401 tablet_bootstrap.cc:654] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:48.688879 8400 log.cc:824] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:48.689545 8402 tablet_bootstrap.cc:492] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c: Bootstrap starting.
I20260620 14:57:48.689683 8401 log.cc:824] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:48.691228 8401 tablet_bootstrap.cc:492] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b: No bootstrap required, opened a new log
I20260620 14:57:48.691298 8400 tablet_bootstrap.cc:492] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5: No bootstrap required, opened a new log
I20260620 14:57:48.691433 8401 ts_tablet_manager.cc:1397] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:57:48.691478 8400 ts_tablet_manager.cc:1397] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260620 14:57:48.691509 8402 tablet_bootstrap.cc:654] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:48.692688 8402 log.cc:824] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:48.694453 8402 tablet_bootstrap.cc:492] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c: No bootstrap required, opened a new log
I20260620 14:57:48.694736 8402 ts_tablet_manager.cc:1397] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:57:48.694629 8401 raft_consensus.cc:348] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.694933 8401 raft_consensus.cc:374] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:48.694994 8401 raft_consensus.cc:729] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcc64e574f74303a0f4bd483143b51b, State: Initialized, Role: FOLLOWER
I20260620 14:57:48.695329 8400 raft_consensus.cc:348] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.695453 8401 consensus_queue.cc:260] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.695571 8400 raft_consensus.cc:374] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:48.695643 8400 raft_consensus.cc:729] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7319253dfff34395ad709aa7562f74c5, State: Initialized, Role: FOLLOWER
I20260620 14:57:48.696061 8249 heartbeater.cc:503] Master 127.7.46.190:33135 was elected leader, sending a full tablet report...
I20260620 14:57:48.696183 8400 consensus_queue.cc:260] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.696410 8401 ts_tablet_manager.cc:1428] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:57:48.697050 8400 ts_tablet_manager.cc:1428] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260620 14:57:48.697381 8115 heartbeater.cc:503] Master 127.7.46.190:33135 was elected leader, sending a full tablet report...
I20260620 14:57:48.698243 8402 raft_consensus.cc:348] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.698530 8402 raft_consensus.cc:374] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:48.698588 8402 raft_consensus.cc:729] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada254838cf64ffb945369ab6233756c, State: Initialized, Role: FOLLOWER
I20260620 14:57:48.699003 8402 consensus_queue.cc:260] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.699560 8383 heartbeater.cc:503] Master 127.7.46.190:33135 was elected leader, sending a full tablet report...
I20260620 14:57:48.700796 8402 ts_tablet_manager.cc:1428] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 14:57:48.875314 8384 tablet.cc:2369] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:48.917582 8250 tablet.cc:2369] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:57:48.925360 8406 raft_consensus.cc:482] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:48.925554 8406 raft_consensus.cc:504] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.926800 8406 leader_election.cc:288] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7319253dfff34395ad709aa7562f74c5 (127.7.46.129:40925), ada254838cf64ffb945369ab6233756c (127.7.46.131:39571)
I20260620 14:57:48.930912 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.927229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:33497 (local address 127.7.46.129:40925)
0620 14:57:48.927398 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.927404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.927422 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:57:48.927571 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.927574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.927651 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.927730 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:48.928079 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.928722 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.929905 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.930324 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.930415 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.930492 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.930702 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.930780 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:57:48.930917 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.927154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:40925 (local address 127.7.46.130:33497)
0620 14:57:48.927307 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:48.927324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:48.927449 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:48.927750 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:48.927753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:48.927799 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:48.927951 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.927960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.928830 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.928834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.929780 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.929790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.930444 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.930450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.930563 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.930596 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:57:48.930673 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:57:48.930747 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":57}
I20260620 14:57:48.930953 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.927442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:39571 (local address 127.7.46.130:44945)
0620 14:57:48.927798 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:48.927812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:48.927902 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:48.928152 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:48.928155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:48.928172 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:48.928286 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.928293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.929040 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.929042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.929831 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:48.929863 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.930522 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.930526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:48.930628 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.930678 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:57:48.930756 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:57:48.930817 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260620 14:57:48.930979 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.927577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:44945 (local address 127.7.46.131:39571)
0620 14:57:48.927714 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.927720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.927896 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:57:48.927986 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.927992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.928048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.928129 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:48.928392 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.928932 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.929975 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.930390 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.930506 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.930568 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.930789 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.930865 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:57:48.931703 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ec116a56174985934b6dd86d3e4a86" candidate_uuid: "6dcc64e574f74303a0f4bd483143b51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada254838cf64ffb945369ab6233756c" is_pre_election: true
I20260620 14:57:48.931748 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ec116a56174985934b6dd86d3e4a86" candidate_uuid: "6dcc64e574f74303a0f4bd483143b51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7319253dfff34395ad709aa7562f74c5" is_pre_election: true
I20260620 14:57:48.932065 8069 raft_consensus.cc:2454] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dcc64e574f74303a0f4bd483143b51b in term 0.
I20260620 14:57:48.932065 8337 raft_consensus.cc:2454] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dcc64e574f74303a0f4bd483143b51b in term 0.
I20260620 14:57:48.932544 8138 leader_election.cc:302] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dcc64e574f74303a0f4bd483143b51b, ada254838cf64ffb945369ab6233756c; no voters:
I20260620 14:57:48.932783 8406 raft_consensus.cc:2781] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:48.932884 8406 raft_consensus.cc:482] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:48.932940 8406 raft_consensus.cc:3037] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:48.933928 8406 raft_consensus.cc:504] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.934263 8406 leader_election.cc:288] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [CANDIDATE]: Term 1 election: Requested vote from peers 7319253dfff34395ad709aa7562f74c5 (127.7.46.129:40925), ada254838cf64ffb945369ab6233756c (127.7.46.131:39571)
I20260620 14:57:48.934695 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ec116a56174985934b6dd86d3e4a86" candidate_uuid: "6dcc64e574f74303a0f4bd483143b51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7319253dfff34395ad709aa7562f74c5"
I20260620 14:57:48.934705 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ec116a56174985934b6dd86d3e4a86" candidate_uuid: "6dcc64e574f74303a0f4bd483143b51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada254838cf64ffb945369ab6233756c"
I20260620 14:57:48.934830 8069 raft_consensus.cc:3037] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:48.934840 8337 raft_consensus.cc:3037] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:48.935870 8069 raft_consensus.cc:2454] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dcc64e574f74303a0f4bd483143b51b in term 1.
I20260620 14:57:48.935889 8337 raft_consensus.cc:2454] T 70ec116a56174985934b6dd86d3e4a86 P ada254838cf64ffb945369ab6233756c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dcc64e574f74303a0f4bd483143b51b in term 1.
I20260620 14:57:48.936169 8138 leader_election.cc:302] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dcc64e574f74303a0f4bd483143b51b, ada254838cf64ffb945369ab6233756c; no voters:
I20260620 14:57:48.936360 8406 raft_consensus.cc:2781] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:48.936590 8406 raft_consensus.cc:686] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [term 1 LEADER]: Becoming Leader. State: Replica: 6dcc64e574f74303a0f4bd483143b51b, State: Running, Role: LEADER
I20260620 14:57:48.936923 8406 consensus_queue.cc:237] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } }
I20260620 14:57:48.940018 7923 catalog_manager.cc:5495] T 70ec116a56174985934b6dd86d3e4a86 P 6dcc64e574f74303a0f4bd483143b51b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dcc64e574f74303a0f4bd483143b51b (127.7.46.130). New cstate: current_term: 1 leader_uuid: "6dcc64e574f74303a0f4bd483143b51b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319253dfff34395ad709aa7562f74c5" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada254838cf64ffb945369ab6233756c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dcc64e574f74303a0f4bd483143b51b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43323 } health_report { overall_health: HEALTHY } } }
W20260620 14:57:48.946360 8116 tablet.cc:2369] T 70ec116a56174985934b6dd86d3e4a86 P 7319253dfff34395ad709aa7562f74c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:57:48.971297 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.962697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.7.46.129:40925)
0620 14:57:48.962834 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.962838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.963028 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:57:48.963160 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.963162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.963202 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.963274 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.963991 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.964499 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.965328 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.965529 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.965612 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.968197 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.968208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.968223 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.968246 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.969742 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.970197 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.970202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.970204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.970292 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.970528 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.970530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.970531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.970780 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.970931 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.971073 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.971177 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:57:48 dist-test-slave-30t4 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:57:48.997402 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.989502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.7.46.129:40925)
0620 14:57:48.989645 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.989649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.989821 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:57:48.989986 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.989994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.990035 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.990130 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.990792 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.991299 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.992193 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.992411 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.992516 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.994751 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.994754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.994757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.994779 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.996049 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.996410 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.996413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.996414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.996487 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.996704 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.996706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.996707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.996849 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.996946 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.997190 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.997283 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:57:49 dist-test-slave-30t4 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:57:49 dist-test-slave-30t4 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:57:49.021978 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:49.013990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.7.46.129:40925)
0620 14:57:49.014129 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:49.014134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:49.014284 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:57:49.014395 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:49.014398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:49.014438 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:49.014508 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:49.015131 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:49.015615 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:49.016470 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:49.016681 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:49.016759 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:49.019040 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:49.019044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:49.019046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:49.019068 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:49.020391 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:49.020812 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:49.020821 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:49.020822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:49.020892 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:49.021173 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:49.021176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:49.021177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:49.021335 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:49.021468 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:49.021700 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:57:49.021810 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:57:49.023303 8029 tablet_service.cc:2908] Rejecting scan request for tablet 70ec116a56174985934b6dd86d3e4a86: Uninitialized: clean time has not yet been initialized
I20260620 14:57:49.025398 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 7986
I20260620 14:57:49.031432 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 8120
I20260620 14:57:49.036782 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 8254
I20260620 14:57:49.041949 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 7892
2026-06-20T14:57:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2462 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:57:49 dist-test-slave-30t4 krb5kdc[8424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:57:49 dist-test-slave-30t4 krb5kdc[8424](info): set up 2 sockets
Jun 20 14:57:49 dist-test-slave-30t4 krb5kdc[8424](info): commencing operation
krb5kdc: starting...
W20260620 14:57:50.075418 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:57:50 dist-test-slave-30t4 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:57:50.192032 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:45339
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:50.296360 8437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:50.296595 8437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:50.296633 8437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:50.300508 8437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:50.300555 8437 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:57:50.300572 8437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:50.300590 8437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:50.300606 8437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:50.300668 8437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:50.305632 8437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:45339
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:50.306586 8437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:50.307571 8437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:50.315706 8443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:50.315682 8442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:50.315743 8437 server_base.cc:1034] running on GCE node
W20260620 14:57:50.315698 8445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:50.316478 8437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:50.317570 8437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:50.318765 8437 hybrid_clock.cc:648] HybridClock initialized: now 1781967470318749 us; error 36 us; skew 500 ppm
Jun 20 14:57:50 dist-test-slave-30t4 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:50.324837 8437 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:57:50.325695 8437 webserver.cc:460] Webserver started at http://127.7.46.190:40335/ using document root <none> and password file <none>
I20260620 14:57:50.326272 8437 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:50.326362 8437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:50.326581 8437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:50.328675 8437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "e8b02318db9347abb73e97e8b465db28"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "2625bb0e12e8dd7a96c0c1a6a8f1b8db"
server_key_iv: "8c1a6d68d89192b15eca7e9b819c084a"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.329241 8437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "e8b02318db9347abb73e97e8b465db28"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "2625bb0e12e8dd7a96c0c1a6a8f1b8db"
server_key_iv: "8c1a6d68d89192b15eca7e9b819c084a"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.332919 8437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:57:50.335213 8452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:50.336443 8437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:57:50.336580 8437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "e8b02318db9347abb73e97e8b465db28"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "2625bb0e12e8dd7a96c0c1a6a8f1b8db"
server_key_iv: "8c1a6d68d89192b15eca7e9b819c084a"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.336684 8437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:50.354936 8437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:50.357563 8437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:50.357758 8437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:50.365144 8437 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:45339
I20260620 14:57:50.365165 8514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:45339 every 8 connection(s)
I20260620 14:57:50.366003 8437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:50.367394 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 8437
I20260620 14:57:50.367507 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:57:50.367798 7354 external_mini_cluster.cc:1428] Setting key 0c0f912438c2f750bceaeb8c82db92f1
I20260620 14:57:50.369108 8515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:50.375252 8515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28: Bootstrap starting.
Jun 20 14:57:50 dist-test-slave-30t4 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:50.377830 8515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:50.378791 8515 log.cc:824] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:50.381002 8515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28: No bootstrap required, opened a new log
I20260620 14:57:50.382591 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.369064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.7.46.190:45339)
0620 14:57:50.369499 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:50.369515 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:50.369550 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:57:50.370156 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:50.370172 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:50.370475 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:50.370739 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:50.371553 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.372587 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.373706 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.374157 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.374272 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.376933 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:50.376939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:50.376953 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:50.376983 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:50.379526 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.380017 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.380023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.380029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.380169 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.380420 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.380423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.380425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.380889 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:50.381076 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:50.381419 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 14:57:50.381699 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":147,"threads_started":1}
I20260620 14:57:50.384631 8515 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b02318db9347abb73e97e8b465db28" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45339 } }
I20260620 14:57:50.384887 8515 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:50.384949 8515 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b02318db9347abb73e97e8b465db28, State: Initialized, Role: FOLLOWER
I20260620 14:57:50.385385 8515 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b02318db9347abb73e97e8b465db28" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45339 } }
I20260620 14:57:50.385527 8515 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:50.385598 8515 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:50.385697 8515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:50.386539 8515 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b02318db9347abb73e97e8b465db28" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45339 } }
I20260620 14:57:50.386868 8515 leader_election.cc:302] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8b02318db9347abb73e97e8b465db28; no voters:
I20260620 14:57:50.387154 8515 leader_election.cc:288] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:57:50.387509 8520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:50.387794 8520 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [term 1 LEADER]: Becoming Leader. State: Replica: e8b02318db9347abb73e97e8b465db28, State: Running, Role: LEADER
I20260620 14:57:50.388168 8520 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b02318db9347abb73e97e8b465db28" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45339 } }
I20260620 14:57:50.388511 8515 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:50.389889 8522 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8b02318db9347abb73e97e8b465db28. Latest consensus state: current_term: 1 leader_uuid: "e8b02318db9347abb73e97e8b465db28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b02318db9347abb73e97e8b465db28" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45339 } } }
I20260620 14:57:50.389901 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8b02318db9347abb73e97e8b465db28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b02318db9347abb73e97e8b465db28" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45339 } } }
I20260620 14:57:50.390055 8522 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:50.390086 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:50.390875 8533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:50.393893 8533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:50.399045 8533 catalog_manager.cc:1347] Generated new cluster ID: 8f25703780ed40f79d8357e746139d01
I20260620 14:57:50.399107 8533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:57:50.418978 8533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:57:50.419912 8533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:57:50.427599 8533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28: Generated new TSK 0
I20260620 14:57:50.428301 8533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:57:50.461139 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:45339
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:50.577605 8541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:50.577843 8541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:50.577885 8541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:50.581796 8541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:50.581848 8541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:50.581887 8541 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:57:50.581918 8541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:57:50.587044 8541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:50.588028 8541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:50.589051 8541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:50.598722 8547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:50.598791 8541 server_base.cc:1034] running on GCE node
W20260620 14:57:50.598713 8546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:50.598730 8549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:50.599249 8541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:50.599959 8541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:50.601148 8541 hybrid_clock.cc:648] HybridClock initialized: now 1781967470601115 us; error 49 us; skew 500 ppm
Jun 20 14:57:50 dist-test-slave-30t4 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:50.603760 8541 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:57:50.604709 8541 webserver.cc:460] Webserver started at http://127.7.46.129:35737/ using document root <none> and password file <none>
I20260620 14:57:50.605268 8541 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:50.605355 8541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:50.605588 8541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:50.607649 8541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "ca92b5999d67b9e4ede7ac7fc0b0a479"
server_key_iv: "8eb1ed56954be8bb4497ef256cb8d52f"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.608258 8541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "ca92b5999d67b9e4ede7ac7fc0b0a479"
server_key_iv: "8eb1ed56954be8bb4497ef256cb8d52f"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.611826 8541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:57:50.614140 8556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:50.615269 8541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:50.615406 8541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "ca92b5999d67b9e4ede7ac7fc0b0a479"
server_key_iv: "8eb1ed56954be8bb4497ef256cb8d52f"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.615513 8541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:50.626389 8541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:50.629263 8541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:50.629454 8541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:50.630931 8541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:50.630999 8541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:50.631067 8541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:50.631116 8541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:50.641872 8541 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:34925
Jun 20 14:57:50 dist-test-slave-30t4 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:50.641888 8671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:34925 every 8 connection(s)
I20260620 14:57:50.642900 8541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:50.645996 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.631623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:44983 (local address 127.7.46.190:45339)
0620 14:57:50.631818 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:50.631823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:50.633141 (+ 1318us) server_negotiation.cc:407] Connection header received
0620 14:57:50.635876 (+ 2735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:50.635882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:50.635940 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:50.636053 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:50.637234 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.638011 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.639252 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.639504 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.639637 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.642640 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:50.642644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:50.642647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:50.642679 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:50.644293 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.644769 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.644772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.644773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.644847 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.645092 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.645094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.645095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.645260 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:50.645334 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:50.645775 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 14:57:50.645875 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:57:50.646785 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.632495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45339 (local address 127.7.46.129:44983)
0620 14:57:50.632956 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:50.633019 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:50.635584 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:50.636263 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:50.636272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:50.636629 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:50.637090 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:50.637102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.638133 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.638136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:50.639123 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:50.639132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.639679 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.639684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:50.639989 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.640721 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:50.640739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:50.642471 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:50.644412 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:50.644416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:50.644427 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:50.644680 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:50.644905 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:50.644907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:50.644909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:50.645019 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:50.645348 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:50.645353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:50.645578 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 14:57:50.645729 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:57:50.645960 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260620 14:57:50.647548 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 8541
I20260620 14:57:50.647682 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:57:50.647970 7354 external_mini_cluster.cc:1428] Setting key e0b89fb3b74d93cec7cd8655ea9a8e53
I20260620 14:57:50.648509 8672 heartbeater.cc:347] Connected to a master server at 127.7.46.190:45339
I20260620 14:57:50.648676 8672 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:50.649143 8672 heartbeater.cc:511] Master 127.7.46.190:45339 requested a full tablet report, sending...
I20260620 14:57:50.650774 8468 ts_manager.cc:194] Registered new tserver with Master: d6959f4f5a3d49d983bcc0e02dfbb1d0 (127.7.46.129:34925)
I20260620 14:57:50.652465 8468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:44983
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 14:57:50.659647 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.650454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45339 (local address 127.7.46.129:58441)
0620 14:57:50.651362 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:50.651374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:50.651488 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:50.651835 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:50.651837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:50.652029 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:50.652138 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:50.652145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.654022 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.654028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:50.655070 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:50.655085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.655549 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.655555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:50.655677 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.656203 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:50.656223 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:50.656689 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:50.658372 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:50.658375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:50.658376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:50.658563 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:50.658795 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:50.658797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:50.658798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:50.658867 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:50.659249 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:50.659251 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:50.659322 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 14:57:50.659400 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:57:50.659498 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":109,"threads_started":1}
I20260620 14:57:50.659654 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.650524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:58441 (local address 127.7.46.190:45339)
0620 14:57:50.650627 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:50.650631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:50.651450 (+ 819us) server_negotiation.cc:407] Connection header received
0620 14:57:50.651586 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:50.651589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:50.651638 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:50.651710 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:50.652260 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.653856 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.655214 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.655420 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.655495 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.656821 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:50.656824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:50.656826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:50.656847 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:50.658273 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.658646 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.658648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.658650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.658714 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.658978 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.658984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.658989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.659168 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:50.659245 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:50.659455 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:57:50.659539 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":725,"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:57:50.675021 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:45339
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:50.784577 8681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:50.784852 8681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:50.784935 8681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:50.789140 8681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:50.789214 8681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:50.789276 8681 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:57:50.789340 8681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:57:50.794548 8681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:50.795606 8681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:50.796657 8681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:50.806124 8686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:50.806094 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:50.806118 8687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:50.806476 8681 server_base.cc:1034] running on GCE node
I20260620 14:57:50.806886 8681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:50.807512 8681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:50.808687 8681 hybrid_clock.cc:648] HybridClock initialized: now 1781967470808668 us; error 30 us; skew 500 ppm
Jun 20 14:57:50 dist-test-slave-30t4 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:50.811604 8681 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:57:50.812517 8681 webserver.cc:460] Webserver started at http://127.7.46.130:44633/ using document root <none> and password file <none>
I20260620 14:57:50.813083 8681 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:50.813143 8681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:50.813373 8681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:50.815394 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "6d79c0c9a52b4040990e31ab296c7396"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "bb069f309b243dc5d50b8882ba2d2acd"
server_key_iv: "8fe084278c820f964d8e78f633b71413"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.815989 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "6d79c0c9a52b4040990e31ab296c7396"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "bb069f309b243dc5d50b8882ba2d2acd"
server_key_iv: "8fe084278c820f964d8e78f633b71413"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.819878 8681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:57:50.822351 8696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:50.823616 8681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:50.823753 8681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "6d79c0c9a52b4040990e31ab296c7396"
format_stamp: "Formatted at 2026-06-20 14:57:50 on dist-test-slave-30t4"
server_key: "bb069f309b243dc5d50b8882ba2d2acd"
server_key_iv: "8fe084278c820f964d8e78f633b71413"
server_key_version: "encryptionkey@0"
I20260620 14:57:50.823925 8681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:50.835230 8681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:50.838364 8681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:50.838557 8681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:50.840056 8681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:50.840130 8681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:50.840202 8681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:50.840245 8681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:57:50 dist-test-slave-30t4 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:50.851080 8681 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:44773
I20260620 14:57:50.851150 8811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:44773 every 8 connection(s)
I20260620 14:57:50.852178 8681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:57:50.854198 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.840927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:37119 (local address 127.7.46.190:45339)
0620 14:57:50.841099 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:50.841103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:50.842137 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 14:57:50.843215 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:50.843219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:50.843287 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:50.843386 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:50.844883 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.845797 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.847258 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.847585 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.847697 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.850549 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:50.850553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:50.850556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:50.850588 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:50.852206 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.852720 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.852725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.852729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.852827 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.853228 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.853230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.853232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.853410 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:50.853490 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:50.853971 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 14:57:50.854069 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:57:50.855016 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.841504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45339 (local address 127.7.46.130:37119)
0620 14:57:50.841993 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:50.842032 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:50.842966 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:50.843612 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:50.843621 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:50.844105 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:50.844728 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:50.844746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.845941 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.845947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:50.847134 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:50.847143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.847748 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.847751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:50.847958 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.848558 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:50.848574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:50.850371 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:50.852324 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:50.852329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:50.852344 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:50.852624 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:50.852927 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:50.852931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:50.852933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:50.853122 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:50.853528 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:50.853533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:50.853764 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 14:57:50.853928 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:57:50.854191 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260620 14:57:50.856194 8812 heartbeater.cc:347] Connected to a master server at 127.7.46.190:45339
I20260620 14:57:50.856326 8812 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:50.856850 8812 heartbeater.cc:511] Master 127.7.46.190:45339 requested a full tablet report, sending...
I20260620 14:57:50.857919 8468 ts_manager.cc:194] Registered new tserver with Master: 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773)
I20260620 14:57:50.858670 8468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:37119
I20260620 14:57:50.860944 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 8681
I20260620 14:57:50.861068 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:57:50.861331 7354 external_mini_cluster.cc:1428] Setting key 912cb51ab10e17efff21a2a8900700e7
I20260620 14:57:50.865654 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.858400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45339 (local address 127.7.46.130:43299)
0620 14:57:50.858684 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:50.858700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:50.858793 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:50.859207 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:50.859209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:50.859365 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:50.859459 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:50.859464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.860264 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.860274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:50.860817 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:50.860824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.861370 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.861374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:50.861469 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.861982 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:50.861997 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:50.862360 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:50.864241 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:50.864245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:50.864248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:50.864438 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:50.864751 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:50.864753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:50.864754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:50.864821 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:50.865293 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:50.865295 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:50.865362 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 14:57:50.865435 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:57:50.865529 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260620 14:57:50.865839 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:50.858475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:43299 (local address 127.7.46.190:45339)
0620 14:57:50.858583 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:50.858589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:50.858814 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:57:50.858862 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:50.858864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:50.858919 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:50.858997 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:50.859570 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.860149 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.860931 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:50.861265 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:50.861369 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:50.862469 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:50.862473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:50.862475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:50.862495 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:50.864115 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.864535 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.864540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.864544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.864652 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:50.864907 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:50.864911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:50.864914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:50.865199 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:50.865297 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:50.865520 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:57:50.865654 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:57:50.885947 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:45339
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:50.992825 8821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:50.993116 8821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:50.993196 8821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:50.997241 8821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:50.997318 8821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:50.997386 8821 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:57:50.997462 8821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:57:51.002666 8821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:51.003697 8821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:51.004706 8821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:51.014070 8829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:51.014127 8827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:51.014070 8826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:51.014245 8821 server_base.cc:1034] running on GCE node
I20260620 14:57:51.014742 8821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:51.015363 8821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:51.016511 8821 hybrid_clock.cc:648] HybridClock initialized: now 1781967471016484 us; error 40 us; skew 500 ppm
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967471, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:51.019238 8821 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:57:51.020167 8821 webserver.cc:460] Webserver started at http://127.7.46.131:41783/ using document root <none> and password file <none>
I20260620 14:57:51.020717 8821 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:51.020807 8821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:51.021034 8821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:51.023033 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "59d621c0460e4960a5f1b95f20fd70bb"
format_stamp: "Formatted at 2026-06-20 14:57:51 on dist-test-slave-30t4"
server_key: "1d499ce22db2036f4b9fb8b28975830b"
server_key_iv: "acc0638f0d33833b7944d3650dc9c236"
server_key_version: "encryptionkey@0"
I20260620 14:57:51.023599 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "59d621c0460e4960a5f1b95f20fd70bb"
format_stamp: "Formatted at 2026-06-20 14:57:51 on dist-test-slave-30t4"
server_key: "1d499ce22db2036f4b9fb8b28975830b"
server_key_iv: "acc0638f0d33833b7944d3650dc9c236"
server_key_version: "encryptionkey@0"
I20260620 14:57:51.027263 8821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:57:51.029747 8836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:51.031018 8821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:51.031204 8821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "59d621c0460e4960a5f1b95f20fd70bb"
format_stamp: "Formatted at 2026-06-20 14:57:51 on dist-test-slave-30t4"
server_key: "1d499ce22db2036f4b9fb8b28975830b"
server_key_iv: "acc0638f0d33833b7944d3650dc9c236"
server_key_version: "encryptionkey@0"
I20260620 14:57:51.031317 8821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:51.050724 8821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:51.053666 8821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:51.053874 8821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:51.055281 8821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:51.055364 8821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:51.055436 8821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:51.055481 8821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967471, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:51.066378 8821 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:35427
I20260620 14:57:51.066406 8951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:35427 every 8 connection(s)
I20260620 14:57:51.067229 8821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:57:51.070020 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.056316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:60023 (local address 127.7.46.190:45339)
0620 14:57:51.056445 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.056448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.057595 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 14:57:51.058662 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.058665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.058726 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.058813 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:51.060467 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.061336 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.063191 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.063438 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.063525 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.066428 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:51.066453 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:51.066457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:51.066489 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:51.068172 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.068597 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.068599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.068600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.068671 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.068946 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.068948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.068949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.069110 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:51.069183 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.069594 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.069879 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:57:51.070596 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.056961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45339 (local address 127.7.46.131:60023)
0620 14:57:51.057447 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.057489 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.058443 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.059029 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.059041 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.059529 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:51.060283 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.060307 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.061455 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.061459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.063055 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.063069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.063596 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.063602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.063900 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.064616 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:51.064635 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:51.066229 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:51.068256 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:51.068261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:51.068271 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:51.068504 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:51.068753 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:51.068755 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:51.068757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:51.068871 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:51.069189 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:51.069193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:51.069395 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.069543 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.069782 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":136,"threads_started":1}
I20260620 14:57:51.071563 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 8821
I20260620 14:57:51.071693 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:57:51.071991 8952 heartbeater.cc:347] Connected to a master server at 127.7.46.190:45339
I20260620 14:57:51.072022 7354 external_mini_cluster.cc:1428] Setting key 3763b6c80798294561b59298a35fa921
I20260620 14:57:51.072139 8952 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:51.072686 8952 heartbeater.cc:511] Master 127.7.46.190:45339 requested a full tablet report, sending...
I20260620 14:57:51.073655 8468 ts_manager.cc:194] Registered new tserver with Master: 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131:35427)
I20260620 14:57:51.074366 8468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:60023
I20260620 14:57:51.080163 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.073901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45339 (local address 127.7.46.131:55539)
0620 14:57:51.074254 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.074265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.074342 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.074561 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.074563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.074768 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:51.074858 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.074863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.075562 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.075565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.076350 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.076358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.076744 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.076746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.076809 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.077316 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:51.077327 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:51.077604 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:51.078949 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:51.078951 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:51.078953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:51.079134 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:51.079359 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:51.079360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:51.079362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:51.079428 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:51.079731 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:51.079733 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:51.079850 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.079931 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.080027 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":113,"threads_started":1}
I20260620 14:57:51.080178 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.073990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:55539 (local address 127.7.46.190:45339)
0620 14:57:51.074121 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.074127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.074340 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:57:51.074405 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.074408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.074451 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.074543 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:51.074970 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.075439 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.076447 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.076654 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.076730 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.077709 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:51.077712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:51.077714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:51.077730 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:51.078857 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.079213 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.079216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.079217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.079284 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.079500 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.079502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.079503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.079656 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:51.079727 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.079992 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.080074 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:57:51.085546 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:51.109984 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.101736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.7.46.190:45339)
0620 14:57:51.101876 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.101879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.101891 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:57:51.101924 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.101926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.101961 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.102049 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:51.102716 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.103303 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.104379 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.104604 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.104716 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.106976 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:51.106984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:51.106986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:51.107009 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:51.108397 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.108799 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.108803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.108805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.108882 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.109152 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.109156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.109158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.109326 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:51.109470 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.109692 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.109803 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:57:51.112864 8468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:57:51.115974 8468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:51.130227 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.125193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.7.46.129:34925)
0620 14:57:51.125729 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.125742 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.125983 (+ 241us) server_negotiation.cc:407] Connection header received
0620 14:57:51.126084 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.126089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.126301 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.126478 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.126600 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.127686 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.128965 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.129573 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.129698 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.129868 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.129986 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.130074 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":67,"threads_started":1}
I20260620 14:57:51.130218 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.124770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:34925 (local address 127.0.0.1:46284)
0620 14:57:51.125871 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.125883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.125982 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.126439 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.126442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.126455 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.126517 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.126521 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.127819 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.127822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.128844 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.128855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.129699 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.129702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.129805 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.129842 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.129959 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.130074 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":119,"thread_start_us":90,"threads_started":1}
I20260620 14:57:51.130455 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.124024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35427 (local address 127.0.0.1:56534)
0620 14:57:51.124374 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.124423 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.124540 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.125574 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.125581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.125606 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.125737 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.125745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.127132 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.127136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.128632 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.128643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.129942 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.129946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.130050 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.130074 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.130140 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.130216 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260620 14:57:51.130846 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.124170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.7.46.131:35427)
0620 14:57:51.124870 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.124883 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.124909 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:57:51.124985 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.124991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.125219 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.125384 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.125874 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.127019 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.128773 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.129578 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.129708 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.129872 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.130298 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.130707 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":107,"threads_started":1}
I20260620 14:57:51.130860 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.124437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:44773 (local address 127.0.0.1:48892)
0620 14:57:51.124996 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.125008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.125088 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.126189 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.126192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.126210 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.126296 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.126301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.127721 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.127727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.129051 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.129060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.130419 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.130425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.130596 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.130619 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.130686 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.130745 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":114,"thread_start_us":91,"threads_started":1}
I20260620 14:57:51.131310 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.125415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.7.46.130:44773)
0620 14:57:51.125719 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.125736 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.125759 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:57:51.125834 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.125841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.126055 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.126207 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.126412 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.127573 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.129180 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.129963 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.130058 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.130217 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.131075 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.131179 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260620 14:57:51.133051 8606 tablet_service.cc:1467] Processing CreateTablet for tablet f63c6ffbc73845788c1b06d0a3c88b3e (DEFAULT_TABLE table=test-table [id=e3eaf8e7c94048e1872376096a2b3d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:51.133052 8886 tablet_service.cc:1467] Processing CreateTablet for tablet f63c6ffbc73845788c1b06d0a3c88b3e (DEFAULT_TABLE table=test-table [id=e3eaf8e7c94048e1872376096a2b3d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:51.133952 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63c6ffbc73845788c1b06d0a3c88b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.133961 8886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63c6ffbc73845788c1b06d0a3c88b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.137777 8746 tablet_service.cc:1467] Processing CreateTablet for tablet f63c6ffbc73845788c1b06d0a3c88b3e (DEFAULT_TABLE table=test-table [id=e3eaf8e7c94048e1872376096a2b3d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:51.138761 8746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63c6ffbc73845788c1b06d0a3c88b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.140694 8975 tablet_bootstrap.cc:492] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: Bootstrap starting.
I20260620 14:57:51.141700 8976 tablet_bootstrap.cc:492] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: Bootstrap starting.
I20260620 14:57:51.142812 8975 tablet_bootstrap.cc:654] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.143527 8975 log.cc:824] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:51.143744 8976 tablet_bootstrap.cc:654] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.144572 8976 log.cc:824] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:51.145574 8975 tablet_bootstrap.cc:492] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: No bootstrap required, opened a new log
I20260620 14:57:51.145802 8975 ts_tablet_manager.cc:1397] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:57:51.146157 8979 tablet_bootstrap.cc:492] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: Bootstrap starting.
I20260620 14:57:51.146361 8976 tablet_bootstrap.cc:492] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: No bootstrap required, opened a new log
I20260620 14:57:51.146536 8976 ts_tablet_manager.cc:1397] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:57:51.148417 8979 tablet_bootstrap.cc:654] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.149210 8979 log.cc:824] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:51.149672 8976 raft_consensus.cc:348] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.149746 8975 raft_consensus.cc:348] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.149946 8976 raft_consensus.cc:374] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.149955 8975 raft_consensus.cc:374] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.150000 8976 raft_consensus.cc:729] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6959f4f5a3d49d983bcc0e02dfbb1d0, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.150007 8975 raft_consensus.cc:729] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d621c0460e4960a5f1b95f20fd70bb, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.150427 8976 consensus_queue.cc:260] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.150465 8975 consensus_queue.cc:260] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.150992 8672 heartbeater.cc:503] Master 127.7.46.190:45339 was elected leader, sending a full tablet report...
I20260620 14:57:51.151105 8979 tablet_bootstrap.cc:492] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: No bootstrap required, opened a new log
I20260620 14:57:51.151283 8979 ts_tablet_manager.cc:1397] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:57:51.151357 8976 ts_tablet_manager.cc:1428] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260620 14:57:51.151389 8975 ts_tablet_manager.cc:1428] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:57:51.151952 8952 heartbeater.cc:503] Master 127.7.46.190:45339 was elected leader, sending a full tablet report...
I20260620 14:57:51.154830 8979 raft_consensus.cc:348] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.155090 8979 raft_consensus.cc:374] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.155184 8979 raft_consensus.cc:729] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d79c0c9a52b4040990e31ab296c7396, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.155647 8979 consensus_queue.cc:260] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.156280 8812 heartbeater.cc:503] Master 127.7.46.190:45339 was elected leader, sending a full tablet report...
I20260620 14:57:51.156507 8979 ts_tablet_manager.cc:1428] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:57:51.166661 8981 raft_consensus.cc:482] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:51.166836 8981 raft_consensus.cc:504] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.167862 8981 leader_election.cc:288] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131:35427), 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773)
I20260620 14:57:51.172300 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.168127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:32901 (local address 127.7.46.131:35427)
0620 14:57:51.168254 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.168259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.168394 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:57:51.168546 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.168549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.168592 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.168673 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.169070 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.169861 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.171255 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.171662 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.171837 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.171890 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.172106 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.172178 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:57:51.172361 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.168019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35427 (local address 127.7.46.129:32901)
0620 14:57:51.168318 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.168333 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.168445 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.168700 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.168703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.168742 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.168954 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.168961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.169969 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.169972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.171133 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.171143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.171837 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.171840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.171981 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.172001 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.172091 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.172170 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":161,"spinlock_wait_cycles":37248,"thread_start_us":79,"threads_started":1}
I20260620 14:57:51.172359 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.168225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:35353 (local address 127.7.46.130:44773)
0620 14:57:51.168347 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.168351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.168463 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:57:51.168536 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.168539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.168586 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.168673 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.169070 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.169856 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.171257 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.171810 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.171926 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.171981 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.172182 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.172245 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:57:51.172389 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.168103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:44773 (local address 127.7.46.129:35353)
0620 14:57:51.168370 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.168384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.168456 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.168700 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.168703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.168743 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.168954 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.168961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.170018 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.170021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.171138 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.171148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.171915 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.171919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.172053 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.172082 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.172167 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.172236 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":75,"spinlock_wait_cycles":3968,"thread_start_us":71,"threads_started":1}
I20260620 14:57:51.172966 8766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63c6ffbc73845788c1b06d0a3c88b3e" candidate_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d79c0c9a52b4040990e31ab296c7396" is_pre_election: true
I20260620 14:57:51.173108 8906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63c6ffbc73845788c1b06d0a3c88b3e" candidate_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d621c0460e4960a5f1b95f20fd70bb" is_pre_election: true
I20260620 14:57:51.173270 8766 raft_consensus.cc:2454] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6959f4f5a3d49d983bcc0e02dfbb1d0 in term 0.
I20260620 14:57:51.173381 8906 raft_consensus.cc:2454] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6959f4f5a3d49d983bcc0e02dfbb1d0 in term 0.
I20260620 14:57:51.173758 8560 leader_election.cc:302] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d79c0c9a52b4040990e31ab296c7396, d6959f4f5a3d49d983bcc0e02dfbb1d0; no voters:
I20260620 14:57:51.174041 8981 raft_consensus.cc:2781] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:51.174147 8981 raft_consensus.cc:482] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:51.174199 8981 raft_consensus.cc:3037] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.175397 8981 raft_consensus.cc:504] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.175835 8981 leader_election.cc:288] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [CANDIDATE]: Term 1 election: Requested vote from peers 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131:35427), 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773)
I20260620 14:57:51.176192 8906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63c6ffbc73845788c1b06d0a3c88b3e" candidate_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d621c0460e4960a5f1b95f20fd70bb"
I20260620 14:57:51.176321 8906 raft_consensus.cc:3037] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.176889 8766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63c6ffbc73845788c1b06d0a3c88b3e" candidate_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d79c0c9a52b4040990e31ab296c7396"
I20260620 14:57:51.177003 8766 raft_consensus.cc:3037] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.177670 8906 raft_consensus.cc:2454] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6959f4f5a3d49d983bcc0e02dfbb1d0 in term 1.
I20260620 14:57:51.177974 8559 leader_election.cc:302] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d621c0460e4960a5f1b95f20fd70bb, d6959f4f5a3d49d983bcc0e02dfbb1d0; no voters:
I20260620 14:57:51.178197 8981 raft_consensus.cc:2781] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:51.178328 8766 raft_consensus.cc:2454] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6959f4f5a3d49d983bcc0e02dfbb1d0 in term 1.
I20260620 14:57:51.178442 8981 raft_consensus.cc:686] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 LEADER]: Becoming Leader. State: Replica: d6959f4f5a3d49d983bcc0e02dfbb1d0, State: Running, Role: LEADER
I20260620 14:57:51.178802 8981 consensus_queue.cc:237] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } }
I20260620 14:57:51.181807 8469 catalog_manager.cc:5495] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6959f4f5a3d49d983bcc0e02dfbb1d0 (127.7.46.129). New cstate: current_term: 1 leader_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } health_report { overall_health: UNKNOWN } } }
I20260620 14:57:51.212024 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.208068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.7.46.129:34925)
0620 14:57:51.208227 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.208232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.208247 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:57:51.208469 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.208471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.208531 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.208631 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:51.208942 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.209443 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.210365 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.210685 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.210834 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.211187 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:51.211573 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:51.211698 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.211810 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.211870 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:57:51.218618 8766 raft_consensus.cc:1261] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Refusing update from remote peer d6959f4f5a3d49d983bcc0e02dfbb1d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:51.219362 8986 consensus_queue.cc:1035] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:51.219861 8906 raft_consensus.cc:1261] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Refusing update from remote peer d6959f4f5a3d49d983bcc0e02dfbb1d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:51.220484 8986 consensus_queue.cc:1035] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:51.229354 8990 mvcc.cc:205] Tried to move back new op lower bound from 7298938762103685120 to 7298938761950973952. Current Snapshot: MvccSnapshot[applied={T|T < 7298938762103685120}]
I20260620 14:57:51.239670 8468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:51.239863 8468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:51.242307 8468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28: Sending DeleteTablet for 3 replicas of tablet f63c6ffbc73845788c1b06d0a3c88b3e
I20260620 14:57:51.243090 8886 tablet_service.cc:1514] Processing DeleteTablet for tablet f63c6ffbc73845788c1b06d0a3c88b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:51 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:56534
I20260620 14:57:51.243191 8746 tablet_service.cc:1514] Processing DeleteTablet for tablet f63c6ffbc73845788c1b06d0a3c88b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:51 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:48892
I20260620 14:57:51.243569 8999 tablet_replica.cc:330] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: stopping tablet replica
I20260620 14:57:51.243628 8606 tablet_service.cc:1514] Processing DeleteTablet for tablet f63c6ffbc73845788c1b06d0a3c88b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:51 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:46284
I20260620 14:57:51.243929 8999 raft_consensus.cc:2229] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:51.244199 9000 tablet_replica.cc:330] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: stopping tablet replica
I20260620 14:57:51.244241 8999 raft_consensus.cc:2258] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:51.244179 8468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:57:51.244455 9000 raft_consensus.cc:2229] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:57:51.244561 8468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:51.244716 9000 raft_consensus.cc:2258] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:51.245024 9001 tablet_replica.cc:330] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: stopping tablet replica
I20260620 14:57:51.245265 9001 raft_consensus.cc:2229] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:51.245616 9001 raft_consensus.cc:2258] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:51.246984 9001 ts_tablet_manager.cc:1905] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:51.248351 8999 ts_tablet_manager.cc:1905] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:51.249313 8746 tablet_service.cc:1467] Processing CreateTablet for tablet 94e8bd3bb54047d99abf321ca74bf88d (DEFAULT_TABLE table=test-table [id=0d2e5f3df2494cafa36c99265f31fb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:51.249399 8606 tablet_service.cc:1467] Processing CreateTablet for tablet 94e8bd3bb54047d99abf321ca74bf88d (DEFAULT_TABLE table=test-table [id=0d2e5f3df2494cafa36c99265f31fb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:51.249560 8746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e8bd3bb54047d99abf321ca74bf88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.249634 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e8bd3bb54047d99abf321ca74bf88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.250010 9000 ts_tablet_manager.cc:1905] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:51.251989 8976 tablet_bootstrap.cc:492] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: Bootstrap starting.
I20260620 14:57:51.252022 8979 tablet_bootstrap.cc:492] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: Bootstrap starting.
I20260620 14:57:51.252503 9000 ts_tablet_manager.cc:1918] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:51.252592 9000 log.cc:1196] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/f63c6ffbc73845788c1b06d0a3c88b3e
I20260620 14:57:51.252920 8979 tablet_bootstrap.cc:654] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.252954 9000 ts_tablet_manager.cc:1939] T f63c6ffbc73845788c1b06d0a3c88b3e P 6d79c0c9a52b4040990e31ab296c7396: Deleting consensus metadata
I20260620 14:57:51.253041 8976 tablet_bootstrap.cc:654] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.254148 8456 catalog_manager.cc:4841] TS 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773): tablet f63c6ffbc73845788c1b06d0a3c88b3e (table test-table [id=e3eaf8e7c94048e1872376096a2b3d0b]) successfully deleted
I20260620 14:57:51.253624 8886 tablet_service.cc:1467] Processing CreateTablet for tablet 94e8bd3bb54047d99abf321ca74bf88d (DEFAULT_TABLE table=test-table [id=0d2e5f3df2494cafa36c99265f31fb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:51.254498 8886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e8bd3bb54047d99abf321ca74bf88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.254747 9001 ts_tablet_manager.cc:1918] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:51.254832 9001 log.cc:1196] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/f63c6ffbc73845788c1b06d0a3c88b3e
I20260620 14:57:51.255122 9001 ts_tablet_manager.cc:1939] T f63c6ffbc73845788c1b06d0a3c88b3e P d6959f4f5a3d49d983bcc0e02dfbb1d0: Deleting consensus metadata
I20260620 14:57:51.255707 8976 tablet_bootstrap.cc:492] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: No bootstrap required, opened a new log
I20260620 14:57:51.255817 8976 ts_tablet_manager.cc:1397] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 14:57:51.256160 8455 catalog_manager.cc:4841] TS d6959f4f5a3d49d983bcc0e02dfbb1d0 (127.7.46.129:34925): tablet f63c6ffbc73845788c1b06d0a3c88b3e (table test-table [id=e3eaf8e7c94048e1872376096a2b3d0b]) successfully deleted
I20260620 14:57:51.256314 8976 raft_consensus.cc:348] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.256413 8976 raft_consensus.cc:374] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.256443 8976 raft_consensus.cc:729] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6959f4f5a3d49d983bcc0e02dfbb1d0, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.256549 8976 consensus_queue.cc:260] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.256834 8976 ts_tablet_manager.cc:1428] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:51.256934 8979 tablet_bootstrap.cc:492] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: No bootstrap required, opened a new log
I20260620 14:57:51.257023 8979 ts_tablet_manager.cc:1397] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 14:57:51.257426 8979 raft_consensus.cc:348] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.257515 8979 raft_consensus.cc:374] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.257544 8979 raft_consensus.cc:729] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d79c0c9a52b4040990e31ab296c7396, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.257584 8975 tablet_bootstrap.cc:492] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: Bootstrap starting.
I20260620 14:57:51.257648 8979 consensus_queue.cc:260] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.257862 8979 ts_tablet_manager.cc:1428] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:51.258556 8975 tablet_bootstrap.cc:654] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.259444 8999 ts_tablet_manager.cc:1918] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:51.259532 8999 log.cc:1196] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/f63c6ffbc73845788c1b06d0a3c88b3e
I20260620 14:57:51.259873 8999 ts_tablet_manager.cc:1939] T f63c6ffbc73845788c1b06d0a3c88b3e P 59d621c0460e4960a5f1b95f20fd70bb: Deleting consensus metadata
I20260620 14:57:51.260941 8975 tablet_bootstrap.cc:492] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: No bootstrap required, opened a new log
I20260620 14:57:51.260974 8455 catalog_manager.cc:4841] TS 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131:35427): tablet f63c6ffbc73845788c1b06d0a3c88b3e (table test-table [id=e3eaf8e7c94048e1872376096a2b3d0b]) successfully deleted
I20260620 14:57:51.261023 8975 ts_tablet_manager.cc:1397] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:51.261521 8975 raft_consensus.cc:348] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.261626 8975 raft_consensus.cc:374] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.261649 8975 raft_consensus.cc:729] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d621c0460e4960a5f1b95f20fd70bb, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.261770 8975 consensus_queue.cc:260] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.262002 8975 ts_tablet_manager.cc:1428] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260620 14:57:51.318603 8953 tablet.cc:2369] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:51.353363 8813 tablet.cc:2369] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:51.394222 8673 tablet.cc:2369] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:57:51.571259 8996 raft_consensus.cc:482] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:51.571442 8996 raft_consensus.cc:504] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.571993 8996 leader_election.cc:288] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773), 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131:35427)
I20260620 14:57:51.572484 8906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e8bd3bb54047d99abf321ca74bf88d" candidate_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d621c0460e4960a5f1b95f20fd70bb" is_pre_election: true
I20260620 14:57:51.572507 8766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e8bd3bb54047d99abf321ca74bf88d" candidate_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d79c0c9a52b4040990e31ab296c7396" is_pre_election: true
I20260620 14:57:51.572638 8906 raft_consensus.cc:2454] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6959f4f5a3d49d983bcc0e02dfbb1d0 in term 0.
I20260620 14:57:51.572676 8766 raft_consensus.cc:2454] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6959f4f5a3d49d983bcc0e02dfbb1d0 in term 0.
I20260620 14:57:51.572968 8560 leader_election.cc:302] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 59d621c0460e4960a5f1b95f20fd70bb, 6d79c0c9a52b4040990e31ab296c7396, d6959f4f5a3d49d983bcc0e02dfbb1d0; no voters:
I20260620 14:57:51.573184 8996 raft_consensus.cc:2781] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:51.573256 8996 raft_consensus.cc:482] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:51.573310 8996 raft_consensus.cc:3037] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.574295 8996 raft_consensus.cc:504] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.574608 8996 leader_election.cc:288] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [CANDIDATE]: Term 1 election: Requested vote from peers 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773), 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131:35427)
I20260620 14:57:51.574971 8766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e8bd3bb54047d99abf321ca74bf88d" candidate_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d79c0c9a52b4040990e31ab296c7396"
I20260620 14:57:51.574985 8906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e8bd3bb54047d99abf321ca74bf88d" candidate_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d621c0460e4960a5f1b95f20fd70bb"
I20260620 14:57:51.575062 8766 raft_consensus.cc:3037] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.575095 8906 raft_consensus.cc:3037] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.575953 8906 raft_consensus.cc:2454] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6959f4f5a3d49d983bcc0e02dfbb1d0 in term 1.
I20260620 14:57:51.576098 8766 raft_consensus.cc:2454] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6959f4f5a3d49d983bcc0e02dfbb1d0 in term 1.
I20260620 14:57:51.576241 8559 leader_election.cc:302] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d621c0460e4960a5f1b95f20fd70bb, d6959f4f5a3d49d983bcc0e02dfbb1d0; no voters:
I20260620 14:57:51.576411 8996 raft_consensus.cc:2781] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:51.576521 8996 raft_consensus.cc:686] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 LEADER]: Becoming Leader. State: Replica: d6959f4f5a3d49d983bcc0e02dfbb1d0, State: Running, Role: LEADER
I20260620 14:57:51.576659 8996 consensus_queue.cc:237] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.578236 8468 catalog_manager.cc:5495] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6959f4f5a3d49d983bcc0e02dfbb1d0 (127.7.46.129). New cstate: current_term: 1 leader_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:51.658556 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.649483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.7.46.190:45339)
0620 14:57:51.649718 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.649722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.649824 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:57:51.649959 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.649961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.650002 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.650078 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:51.650824 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.651364 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.652510 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.652737 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.652821 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.655258 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:51.655261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:51.655263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:51.655281 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:51.656833 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.657338 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.657344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.657346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.657425 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.657693 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.657696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.657697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.657862 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:51.657966 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.658244 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.658405 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260620 14:57:51.658556 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.649476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45339 (local address 127.0.0.1:35694)
0620 14:57:51.649727 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.649743 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.649853 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.650132 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.650135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.650599 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:51.650704 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.650710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.651467 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.651470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.652373 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.652384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.652853 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.652856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.652941 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.653535 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:51.653549 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:51.655116 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:51.656967 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:51.656975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:51.656981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:51.657241 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:51.657515 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:51.657519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:51.657521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:51.657601 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:51.657966 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:51.657974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:51.658084 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.658198 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.658346 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"spinlock_wait_cycles":7552,"thread_start_us":69,"threads_started":1}
I20260620 14:57:51.662367 8468 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:35694:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:57:51.663048 8468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:51.667631 8606 tablet_service.cc:1467] Processing CreateTablet for tablet bc16eb0cb1b84c658c948a0ce639c945 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=146aec9388a844278fd921fc3561086b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:57:51.667725 8746 tablet_service.cc:1467] Processing CreateTablet for tablet bc16eb0cb1b84c658c948a0ce639c945 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=146aec9388a844278fd921fc3561086b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:57:51.668049 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc16eb0cb1b84c658c948a0ce639c945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.668133 8746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc16eb0cb1b84c658c948a0ce639c945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.668155 8886 tablet_service.cc:1467] Processing CreateTablet for tablet bc16eb0cb1b84c658c948a0ce639c945 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=146aec9388a844278fd921fc3561086b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:57:51.668514 8886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc16eb0cb1b84c658c948a0ce639c945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:51.670468 8979 tablet_bootstrap.cc:492] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: Bootstrap starting.
I20260620 14:57:51.670821 8976 tablet_bootstrap.cc:492] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: Bootstrap starting.
I20260620 14:57:51.671304 8975 tablet_bootstrap.cc:492] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: Bootstrap starting.
I20260620 14:57:51.671361 8979 tablet_bootstrap.cc:654] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.671663 8976 tablet_bootstrap.cc:654] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.672366 8975 tablet_bootstrap.cc:654] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:51.673332 8976 tablet_bootstrap.cc:492] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: No bootstrap required, opened a new log
I20260620 14:57:51.673426 8976 ts_tablet_manager.cc:1397] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:57:51.673586 8975 tablet_bootstrap.cc:492] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: No bootstrap required, opened a new log
I20260620 14:57:51.673642 8979 tablet_bootstrap.cc:492] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: No bootstrap required, opened a new log
I20260620 14:57:51.673678 8975 ts_tablet_manager.cc:1397] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:57:51.673708 8979 ts_tablet_manager.cc:1397] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:57:51.673859 8976 raft_consensus.cc:348] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.673974 8976 raft_consensus.cc:374] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.674014 8976 raft_consensus.cc:729] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6959f4f5a3d49d983bcc0e02dfbb1d0, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.674103 8975 raft_consensus.cc:348] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.674090 8979 raft_consensus.cc:348] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.674203 8979 raft_consensus.cc:374] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.674203 8975 raft_consensus.cc:374] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:51.674155 8976 consensus_queue.cc:260] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.674243 8975 raft_consensus.cc:729] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d621c0460e4960a5f1b95f20fd70bb, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.674243 8979 raft_consensus.cc:729] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d79c0c9a52b4040990e31ab296c7396, State: Initialized, Role: FOLLOWER
I20260620 14:57:51.674356 8975 consensus_queue.cc:260] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.674356 8979 consensus_queue.cc:260] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.674427 8976 ts_tablet_manager.cc:1428] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:51.674604 8975 ts_tablet_manager.cc:1428] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:57:51.674620 8979 ts_tablet_manager.cc:1428] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:51.674611 8996 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.674773 8996 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:51.674801 8983 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.674821 8982 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.674926 8983 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:51.674926 8982 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:51.748335 8982 raft_consensus.cc:482] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:51.748471 8982 raft_consensus.cc:504] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.749729 8982 leader_election.cc:288] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6959f4f5a3d49d983bcc0e02dfbb1d0 (127.7.46.129:34925), 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773)
I20260620 14:57:51.754506 8983 raft_consensus.cc:482] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:51.754648 8983 raft_consensus.cc:504] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.755229 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.750045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:38391 (local address 127.7.46.129:34925)
0620 14:57:51.750308 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.750314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.750336 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:57:51.750526 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.750529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.750582 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.750682 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.751103 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.751948 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.753761 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.754403 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.754516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.754571 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.755008 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.755088 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260620 14:57:51.755303 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.749957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:34925 (local address 127.7.46.131:38391)
0620 14:57:51.750270 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.750286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.750405 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.750702 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.750705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.750731 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.750988 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.750992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.752057 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.752060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.753637 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.753649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.754654 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.754658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.754830 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.754881 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.754958 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.755105 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":299,"spinlock_wait_cycles":15744,"thread_start_us":77,"threads_started":1}
I20260620 14:57:51.755303 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.749954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:44773 (local address 127.7.46.131:33077)
0620 14:57:51.750270 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.750286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.750405 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.750702 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.750705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.750731 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.750984 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.750991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.752057 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.752060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.753637 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.753649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.754654 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.754658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.754827 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.754880 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.754962 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.755103 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":313,"spinlock_wait_cycles":24320,"thread_start_us":76,"threads_started":1}
I20260620 14:57:51.755599 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.750068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:33077 (local address 127.7.46.130:44773)
0620 14:57:51.750284 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.750289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.750372 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:57:51.750526 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.750529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.750582 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.750682 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.751103 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.751949 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.753761 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.754397 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.754516 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.754571 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.755367 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.755457 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260620 14:57:51.755882 8983 leader_election.cc:288] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6959f4f5a3d49d983bcc0e02dfbb1d0 (127.7.46.129:34925), 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131:35427)
I20260620 14:57:51.756043 8766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc16eb0cb1b84c658c948a0ce639c945" candidate_uuid: "59d621c0460e4960a5f1b95f20fd70bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d79c0c9a52b4040990e31ab296c7396" is_pre_election: true
I20260620 14:57:51.756193 8766 raft_consensus.cc:2454] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d621c0460e4960a5f1b95f20fd70bb in term 0.
I20260620 14:57:51.756179 8625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc16eb0cb1b84c658c948a0ce639c945" candidate_uuid: "59d621c0460e4960a5f1b95f20fd70bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" is_pre_election: true
I20260620 14:57:51.756392 8625 raft_consensus.cc:2454] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d621c0460e4960a5f1b95f20fd70bb in term 0.
I20260620 14:57:51.756569 8840 leader_election.cc:302] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d621c0460e4960a5f1b95f20fd70bb, 6d79c0c9a52b4040990e31ab296c7396; no voters:
I20260620 14:57:51.756886 8982 raft_consensus.cc:2781] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:51.756953 8982 raft_consensus.cc:482] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:51.757004 8982 raft_consensus.cc:3037] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.758345 8982 raft_consensus.cc:504] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.758730 8982 leader_election.cc:288] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [CANDIDATE]: Term 1 election: Requested vote from peers d6959f4f5a3d49d983bcc0e02dfbb1d0 (127.7.46.129:34925), 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773)
I20260620 14:57:51.759011 8625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc16eb0cb1b84c658c948a0ce639c945" candidate_uuid: "59d621c0460e4960a5f1b95f20fd70bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0"
I20260620 14:57:51.759105 8766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc16eb0cb1b84c658c948a0ce639c945" candidate_uuid: "59d621c0460e4960a5f1b95f20fd70bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d79c0c9a52b4040990e31ab296c7396"
I20260620 14:57:51.759132 8625 raft_consensus.cc:3037] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.759217 8766 raft_consensus.cc:3037] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:51.760306 8766 raft_consensus.cc:2454] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d621c0460e4960a5f1b95f20fd70bb in term 1.
I20260620 14:57:51.760344 8625 raft_consensus.cc:2454] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d621c0460e4960a5f1b95f20fd70bb in term 1.
I20260620 14:57:51.760605 8840 leader_election.cc:302] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d621c0460e4960a5f1b95f20fd70bb, 6d79c0c9a52b4040990e31ab296c7396; no voters:
I20260620 14:57:51.760844 8982 raft_consensus.cc:2781] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:51.761008 8982 raft_consensus.cc:686] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [term 1 LEADER]: Becoming Leader. State: Replica: 59d621c0460e4960a5f1b95f20fd70bb, State: Running, Role: LEADER
I20260620 14:57:51.761274 8982 consensus_queue.cc:237] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } }
I20260620 14:57:51.761576 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.756656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:42233 (local address 127.7.46.129:34925)
0620 14:57:51.757295 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.757298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.757308 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:57:51.757365 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.757368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.757421 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.757480 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.757750 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.758438 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.760006 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.760609 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.760701 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.760747 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.761103 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.761471 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571}
I20260620 14:57:51.761734 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.756546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35427 (local address 127.7.46.130:50275)
0620 14:57:51.756888 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.756903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.756991 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.757803 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.757806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.757821 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.757937 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.757942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.758695 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.758698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.760120 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.760132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.761329 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.761332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.761428 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.761449 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.761523 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.761591 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":50,"thread_start_us":73,"threads_started":1}
I20260620 14:57:51.761880 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.756213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:34925 (local address 127.7.46.130:42233)
0620 14:57:51.756837 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.756856 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.757080 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.757514 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.757517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.757538 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:51.757623 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.757628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.758581 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.758584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.759871 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.759882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.760844 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.760848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.760945 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.760976 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.761686 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.761762 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":50,"thread_start_us":70,"threads_started":1}
I20260620 14:57:51.762441 8625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc16eb0cb1b84c658c948a0ce639c945" candidate_uuid: "6d79c0c9a52b4040990e31ab296c7396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" is_pre_election: true
I20260620 14:57:51.762633 8625 raft_consensus.cc:2379] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d79c0c9a52b4040990e31ab296c7396 in current term 1: Already voted for candidate 59d621c0460e4960a5f1b95f20fd70bb in this term.
I20260620 14:57:51.762828 9023 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: TxnStatusTablet state changed. Reason: New leader 59d621c0460e4960a5f1b95f20fd70bb. Latest consensus state: current_term: 1 leader_uuid: "59d621c0460e4960a5f1b95f20fd70bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.762990 9023 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:57:51.763288 9025 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:57:51.763434 9025 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:57:51.763629 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.756806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:50275 (local address 127.7.46.131:35427)
0620 14:57:51.757626 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.757629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.757638 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:57:51.757669 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.757670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.757700 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.757792 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:51.758039 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.758592 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.760568 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.761199 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.761332 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.763303 (+ 1971us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.763333 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.763431 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":58,"threads_started":1}
I20260620 14:57:51.764003 8906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc16eb0cb1b84c658c948a0ce639c945" candidate_uuid: "6d79c0c9a52b4040990e31ab296c7396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d621c0460e4960a5f1b95f20fd70bb" is_pre_election: true
I20260620 14:57:51.763995 8467 catalog_manager.cc:5495] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131). New cstate: current_term: 1 leader_uuid: "59d621c0460e4960a5f1b95f20fd70bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } health_report { overall_health: HEALTHY } } }
I20260620 14:57:51.764390 8699 leader_election.cc:302] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d79c0c9a52b4040990e31ab296c7396; no voters: 59d621c0460e4960a5f1b95f20fd70bb, d6959f4f5a3d49d983bcc0e02dfbb1d0
I20260620 14:57:51.764636 8983 raft_consensus.cc:2726] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:57:51.795917 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.791934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35427 (local address 127.0.0.1:50522)
0620 14:57:51.792085 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.792103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.792204 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.792535 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.792537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.792544 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:57:51.792643 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.792647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.793615 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.793624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.794208 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.794215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.794789 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.794791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.794859 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.794902 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:57:51.795631 (+ 729us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:57:51.795641 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.795684 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.795757 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:57:51.795923 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.792053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.7.46.131:35427)
0620 14:57:51.792166 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.792171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.792185 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:57:51.792352 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.792355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.792419 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.792522 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:51.792747 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.793515 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.794318 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.794681 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.794782 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.795035 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:51.795534 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:51.795651 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.795712 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.795788 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:57:51.798964 8766 raft_consensus.cc:1261] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Refusing update from remote peer 59d621c0460e4960a5f1b95f20fd70bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:51.799088 8625 raft_consensus.cc:1261] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Refusing update from remote peer 59d621c0460e4960a5f1b95f20fd70bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:51.799582 9023 consensus_queue.cc:1035] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:51.799752 8982 consensus_queue.cc:1035] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:51.801152 8983 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: TxnStatusTablet state changed. Reason: New leader 59d621c0460e4960a5f1b95f20fd70bb. Latest consensus state: current_term: 1 leader_uuid: "59d621c0460e4960a5f1b95f20fd70bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.801258 8983 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:51.801962 8996 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: TxnStatusTablet state changed. Reason: New leader 59d621c0460e4960a5f1b95f20fd70bb. Latest consensus state: current_term: 1 leader_uuid: "59d621c0460e4960a5f1b95f20fd70bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.802079 8996 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:51.803074 8982 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59d621c0460e4960a5f1b95f20fd70bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.803188 8982 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:57:51.803284 9028 mvcc.cc:205] Tried to move back new op lower bound from 7298938764483325952 to 7298938764337844224. Current Snapshot: MvccSnapshot[applied={T|T < 7298938764483325952}]
I20260620 14:57:51.803483 8996 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59d621c0460e4960a5f1b95f20fd70bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.803572 8996 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P d6959f4f5a3d49d983bcc0e02dfbb1d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:51.804350 8982 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59d621c0460e4960a5f1b95f20fd70bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.804442 8982 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P 59d621c0460e4960a5f1b95f20fd70bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:57:51.805596 8983 tablet_replica.cc:439] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59d621c0460e4960a5f1b95f20fd70bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6959f4f5a3d49d983bcc0e02dfbb1d0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34925 } } peers { permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 } } peers { permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 } } }
I20260620 14:57:51.805708 8983 tablet_replica.cc:442] T bc16eb0cb1b84c658c948a0ce639c945 P 6d79c0c9a52b4040990e31ab296c7396: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:51.807211 9031 mvcc.cc:205] Tried to move back new op lower bound from 7298938764483325952 to 7298938764337844224. Current Snapshot: MvccSnapshot[applied={T|T < 7298938764483325952}]
I20260620 14:57:51.819428 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.815893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35427 (local address 127.7.46.129:52993)
0620 14:57:51.816182 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.816196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.816289 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.816511 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.816513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.816520 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:57:51.816617 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.816621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.817565 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.817567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.818227 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.818236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.818777 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.818780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.818850 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.818894 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:57:51.819181 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:57:51.819186 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.819236 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.819303 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
I20260620 14:57:51.819446 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.815935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:52993 (local address 127.7.46.131:35427)
0620 14:57:51.816062 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.816067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.816264 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:57:51.816366 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.816369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.816416 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.816495 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:51.816714 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.817465 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.818335 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.818669 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.818770 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.819018 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:51.819098 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:51.819168 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.819255 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.819303 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:57:51.826284 8906 raft_consensus.cc:1261] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Refusing update from remote peer d6959f4f5a3d49d983bcc0e02dfbb1d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:51.826282 8766 raft_consensus.cc:1261] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Refusing update from remote peer d6959f4f5a3d49d983bcc0e02dfbb1d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:51.826699 8986 consensus_queue.cc:1035] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d79c0c9a52b4040990e31ab296c7396" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:51.826835 8996 consensus_queue.cc:1035] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d621c0460e4960a5f1b95f20fd70bb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:51.850028 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.845276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:34925 (local address 127.7.46.131:40487)
0620 14:57:51.845810 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:51.845829 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:51.845978 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:51.846241 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:51.846244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:51.846253 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:57:51.846385 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.846390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.847182 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.847189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.848503 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:51.848515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.849143 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.849147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:51.849267 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.849367 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:57:51.849709 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:57:51.849718 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:57:51.849792 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:57:51.849874 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":92,"threads_started":1}
I20260620 14:57:51.850003 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.845354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:40487 (local address 127.7.46.129:34925)
0620 14:57:51.845472 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.845478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.845895 (+ 417us) server_negotiation.cc:407] Connection header received
0620 14:57:51.846075 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.846079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.846126 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.846221 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:51.846498 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.847059 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.848649 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.849016 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.849139 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.849523 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:51.849606 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:51.849706 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.849801 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.849852 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:57:51.884886 8467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:51.885027 8467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:51.886965 8467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8b02318db9347abb73e97e8b465db28: Sending DeleteTablet for 3 replicas of tablet 94e8bd3bb54047d99abf321ca74bf88d
I20260620 14:57:51.887547 8746 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e8bd3bb54047d99abf321ca74bf88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:51 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:48892
I20260620 14:57:51.887538 8606 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e8bd3bb54047d99abf321ca74bf88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:51 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:46284
I20260620 14:57:51.888095 8886 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e8bd3bb54047d99abf321ca74bf88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:51 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:56534
I20260620 14:57:51.888144 9046 tablet_replica.cc:330] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: stopping tablet replica
I20260620 14:57:51.888443 9046 raft_consensus.cc:2229] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:51.888479 9049 tablet_replica.cc:330] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: stopping tablet replica
I20260620 14:57:51.888608 9049 raft_consensus.cc:2229] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:51.888739 9049 raft_consensus.cc:2258] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:51.888933 9047 tablet_replica.cc:330] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: stopping tablet replica
I20260620 14:57:51.889083 9047 raft_consensus.cc:2229] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:51.889214 9047 raft_consensus.cc:2258] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:51.889683 9049 ts_tablet_manager.cc:1905] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:51.890034 9047 ts_tablet_manager.cc:1905] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:51.889552 9046 raft_consensus.cc:2258] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:51.891147 9046 ts_tablet_manager.cc:1905] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:51.892925 9049 ts_tablet_manager.cc:1918] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:51.893056 9049 log.cc:1196] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/94e8bd3bb54047d99abf321ca74bf88d
I20260620 14:57:51.893438 9049 ts_tablet_manager.cc:1939] T 94e8bd3bb54047d99abf321ca74bf88d P 59d621c0460e4960a5f1b95f20fd70bb: Deleting consensus metadata
I20260620 14:57:51.893990 9046 ts_tablet_manager.cc:1918] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:51.894093 9046 log.cc:1196] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/94e8bd3bb54047d99abf321ca74bf88d
I20260620 14:57:51.894378 9046 ts_tablet_manager.cc:1939] T 94e8bd3bb54047d99abf321ca74bf88d P d6959f4f5a3d49d983bcc0e02dfbb1d0: Deleting consensus metadata
I20260620 14:57:51.894521 8455 catalog_manager.cc:4841] TS 59d621c0460e4960a5f1b95f20fd70bb (127.7.46.131:35427): tablet 94e8bd3bb54047d99abf321ca74bf88d (table test-table [id=0d2e5f3df2494cafa36c99265f31fb29]) successfully deleted
I20260620 14:57:51.895186 8455 catalog_manager.cc:4841] TS d6959f4f5a3d49d983bcc0e02dfbb1d0 (127.7.46.129:34925): tablet 94e8bd3bb54047d99abf321ca74bf88d (table test-table [id=0d2e5f3df2494cafa36c99265f31fb29]) successfully deleted
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:57:51.897831 9047 ts_tablet_manager.cc:1918] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:51.897915 9047 log.cc:1196] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/94e8bd3bb54047d99abf321ca74bf88d
I20260620 14:57:51.898188 9047 ts_tablet_manager.cc:1939] T 94e8bd3bb54047d99abf321ca74bf88d P 6d79c0c9a52b4040990e31ab296c7396: Deleting consensus metadata
I20260620 14:57:51.898904 8456 catalog_manager.cc:4841] TS 6d79c0c9a52b4040990e31ab296c7396 (127.7.46.130:44773): tablet 94e8bd3bb54047d99abf321ca74bf88d (table test-table [id=0d2e5f3df2494cafa36c99265f31fb29]) successfully deleted
I20260620 14:57:51.900928 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.889197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.7.46.129:34925)
0620 14:57:51.889303 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.889308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.889798 (+ 490us) server_negotiation.cc:407] Connection header received
0620 14:57:51.889986 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.889990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.890038 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.890096 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:51.891129 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.891855 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.892995 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.893311 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.893387 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.896839 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:51.896851 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:51.896870 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:51.896897 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:51.898992 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.899474 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.899480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.899482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.899612 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.899963 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.899967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.899968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.900213 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:51.900418 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.900599 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.900734 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:57:51.901353 8648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50950
I20260620 14:57:51.910452 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:51.903018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.7.46.190:45339)
0620 14:57:51.903143 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:51.903151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:51.903233 (+ 82us) server_negotiation.cc:407] Connection header received
0620 14:57:51.903323 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:51.903326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:51.903377 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:51.903460 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:51.904082 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.904883 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.905626 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:51.905978 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:51.906086 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:51.907297 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:51.907301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:51.907304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:51.907330 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:51.908900 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.909281 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.909284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.909286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.909378 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:51.909644 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:51.909647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:51.909649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:51.909859 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:51.909999 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:51.910179 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:57:51.910298 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:57:51.910868 8467 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35698
I20260620 14:57:51.912055 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 8541
I20260620 14:57:51.922202 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 8681
I20260620 14:57:51.928081 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 8821
I20260620 14:57:51.934689 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 8437
2026-06-20T14:57:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2892 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[9055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[9055](info): set up 2 sockets
Jun 20 14:57:51 dist-test-slave-30t4 krb5kdc[9055](info): commencing operation
krb5kdc: starting...
W20260620 14:57:52.971676 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:57:53.086514 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37871
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:53.194274 9068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:53.194512 9068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:53.194600 9068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:53.198602 9068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:53.198660 9068 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:57:53.198680 9068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:53.198747 9068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:53.198787 9068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:53.198829 9068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:53.204463 9068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:37871
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:53.205474 9068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:53.206476 9068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:53.214910 9073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:53.214888 9074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:53.214977 9068 server_base.cc:1034] running on GCE node
W20260620 14:57:53.214910 9076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:53.215667 9068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:53.216756 9068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:53.217936 9068 hybrid_clock.cc:648] HybridClock initialized: now 1781967473217905 us; error 44 us; skew 500 ppm
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:53.220534 9068 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:57:53.221395 9068 webserver.cc:460] Webserver started at http://127.7.46.190:35429/ using document root <none> and password file <none>
I20260620 14:57:53.221942 9068 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:53.222033 9068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:53.222258 9068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:53.224246 9068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "a13fd78d28dc47af840b2dd4217fbbaf"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "b3eeb069ca23b76cbb8c94f7c1be34d7"
server_key_iv: "350751e12d243337307f7075d784b155"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.224798 9068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "a13fd78d28dc47af840b2dd4217fbbaf"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "b3eeb069ca23b76cbb8c94f7c1be34d7"
server_key_iv: "350751e12d243337307f7075d784b155"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.228308 9068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:57:53.230509 9083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:53.231591 9068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:57:53.231725 9068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "a13fd78d28dc47af840b2dd4217fbbaf"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "b3eeb069ca23b76cbb8c94f7c1be34d7"
server_key_iv: "350751e12d243337307f7075d784b155"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.231890 9068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:53.257789 9068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:53.260426 9068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:53.260634 9068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:53.268410 9068 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:37871
I20260620 14:57:53.268455 9145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:37871 every 8 connection(s)
I20260620 14:57:53.269440 9068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:53.272042 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 9068
I20260620 14:57:53.272087 9146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:53.272172 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:57:53.272410 7354 external_mini_cluster.cc:1428] Setting key 99c49a43e0099d4691a6beddeb941efd
I20260620 14:57:53.278903 9146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: Bootstrap starting.
I20260620 14:57:53.281612 9146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: Neither blocks nor log segments found. Creating new log.
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:53.282496 9146 log.cc:824] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:53.284642 9146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: No bootstrap required, opened a new log
I20260620 14:57:53.287988 9146 raft_consensus.cc:348] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a13fd78d28dc47af840b2dd4217fbbaf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37871 } }
I20260620 14:57:53.288130 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.273735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.7.46.190:37871)
0620 14:57:53.274138 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:53.274156 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:53.274197 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:57:53.275352 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:53.275376 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:53.275811 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:53.276157 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:53.276783 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.278275 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.279138 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.279794 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.279946 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.282995 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:53.283002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:53.283017 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:53.283048 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:53.285355 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.285825 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.285830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.285835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.285973 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.286260 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.286263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.286264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.286623 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:53.286760 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:53.287068 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 14:57:53.287367 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260620 14:57:53.288214 9146 raft_consensus.cc:374] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:53.288281 9146 raft_consensus.cc:729] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13fd78d28dc47af840b2dd4217fbbaf, State: Initialized, Role: FOLLOWER
I20260620 14:57:53.288712 9146 consensus_queue.cc:260] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a13fd78d28dc47af840b2dd4217fbbaf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37871 } }
I20260620 14:57:53.288836 9146 raft_consensus.cc:388] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:53.288921 9146 raft_consensus.cc:482] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:53.289044 9146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:53.289822 9146 raft_consensus.cc:504] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a13fd78d28dc47af840b2dd4217fbbaf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37871 } }
I20260620 14:57:53.290148 9146 leader_election.cc:302] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a13fd78d28dc47af840b2dd4217fbbaf; no voters:
I20260620 14:57:53.290480 9146 leader_election.cc:288] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:57:53.290521 9151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:53.290787 9151 raft_consensus.cc:686] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [term 1 LEADER]: Becoming Leader. State: Replica: a13fd78d28dc47af840b2dd4217fbbaf, State: Running, Role: LEADER
I20260620 14:57:53.291055 9151 consensus_queue.cc:237] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a13fd78d28dc47af840b2dd4217fbbaf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37871 } }
I20260620 14:57:53.291678 9146 sys_catalog.cc:564] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:53.292835 9152 sys_catalog.cc:455] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a13fd78d28dc47af840b2dd4217fbbaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a13fd78d28dc47af840b2dd4217fbbaf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37871 } } }
I20260620 14:57:53.292963 9152 sys_catalog.cc:458] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:53.293022 9153 sys_catalog.cc:455] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a13fd78d28dc47af840b2dd4217fbbaf. Latest consensus state: current_term: 1 leader_uuid: "a13fd78d28dc47af840b2dd4217fbbaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a13fd78d28dc47af840b2dd4217fbbaf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37871 } } }
I20260620 14:57:53.293133 9153 sys_catalog.cc:458] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:53.294010 9160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:53.296643 9160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:53.301544 9160 catalog_manager.cc:1347] Generated new cluster ID: ce4e06c10dce4efb84ea060117a51ba7
I20260620 14:57:53.301620 9160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:57:53.319725 9160 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:57:53.320590 9160 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:57:53.327292 9160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: Generated new TSK 0
I20260620 14:57:53.328071 9160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:57:53.364217 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37871
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:53.471401 9172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:53.471686 9172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:53.471809 9172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:53.475790 9172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:53.475870 9172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:53.475942 9172 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:57:53.475986 9172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:57:53.476049 9172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:57:53.481257 9172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:37871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:53.482257 9172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:53.483230 9172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:53.492575 9180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:53.492616 9172 server_base.cc:1034] running on GCE node
W20260620 14:57:53.492657 9177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:53.492760 9178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:53.493234 9172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:53.493852 9172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:53.495065 9172 hybrid_clock.cc:648] HybridClock initialized: now 1781967473495035 us; error 44 us; skew 500 ppm
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:53.497829 9172 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:57:53.498757 9172 webserver.cc:460] Webserver started at http://127.7.46.129:43125/ using document root <none> and password file <none>
I20260620 14:57:53.499351 9172 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:53.499457 9172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:53.499698 9172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:53.501813 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "efbe10cd8969432bbf558b81bd76c46c"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "49ebe118b6985a8db6fa6179b87f5148"
server_key_iv: "9b2acf590afca41cd01151f337ba8c6d"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.502411 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "efbe10cd8969432bbf558b81bd76c46c"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "49ebe118b6985a8db6fa6179b87f5148"
server_key_iv: "9b2acf590afca41cd01151f337ba8c6d"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.506172 9172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:57:53.508625 9187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:53.509903 9172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:57:53.510001 9172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "efbe10cd8969432bbf558b81bd76c46c"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "49ebe118b6985a8db6fa6179b87f5148"
server_key_iv: "9b2acf590afca41cd01151f337ba8c6d"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.510084 9172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:53.540771 9172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:53.543632 9172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:53.543848 9172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:53.545245 9172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:53.545327 9172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:53.545399 9172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:53.545459 9172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:53.556821 9172 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:42985
I20260620 14:57:53.556893 9302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:42985 every 8 connection(s)
I20260620 14:57:53.557760 9172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:53.559129 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.546063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:43859 (local address 127.7.46.190:37871)
0620 14:57:53.546213 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:53.546218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:53.547024 (+ 806us) server_negotiation.cc:407] Connection header received
0620 14:57:53.548045 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:53.548051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:53.548111 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:53.548172 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:53.549912 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.550916 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.552391 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.552646 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.552759 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.555249 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:53.555256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:53.555261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:53.555298 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:53.557082 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.557560 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.557565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.557567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.557646 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.558008 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.558013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.558015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.558195 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:53.558312 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:53.558876 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 14:57:53.558995 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:57:53.560046 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 9172
I20260620 14:57:53.560129 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.546428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37871 (local address 127.7.46.129:43859)
0620 14:57:53.546853 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:53.546900 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:53.547805 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:53.548608 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:53.548616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:53.549098 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:53.549732 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.549754 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.551066 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.551070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.552270 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.552280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.552806 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.552810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.552973 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.553523 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:53.553542 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:53.555060 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:53.557192 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.557198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.557211 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.557458 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.557737 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.557742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.557744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.557928 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.558308 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:53.558315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:53.558603 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 14:57:53.558825 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 14:57:53.559081 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":120,"threads_started":1}
I20260620 14:57:53.560175 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:57:53.560528 7354 external_mini_cluster.cc:1428] Setting key 63c1cb329cb270a79cd04b5392557b62
I20260620 14:57:53.561595 9303 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37871
I20260620 14:57:53.561728 9303 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:53.562265 9303 heartbeater.cc:511] Master 127.7.46.190:37871 requested a full tablet report, sending...
I20260620 14:57:53.563918 9099 ts_manager.cc:194] Registered new tserver with Master: efbe10cd8969432bbf558b81bd76c46c (127.7.46.129:42985)
I20260620 14:57:53.565140 9099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:43859
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 14:57:53.574957 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.564129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:39033 (local address 127.7.46.190:37871)
0620 14:57:53.564249 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:53.564255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:53.565979 (+ 1724us) server_negotiation.cc:407] Connection header received
0620 14:57:53.566161 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:53.566164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:53.566213 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:53.566267 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:53.566982 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.567758 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.570043 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.570297 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.570416 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.571485 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:53.571491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:53.571496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:53.571526 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:53.573247 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.573692 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.573696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.573698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.573785 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.574085 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.574090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.574093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.574272 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:53.574390 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:53.574685 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 14:57:53.574799 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:57:53.574975 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.564063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37871 (local address 127.7.46.129:39033)
0620 14:57:53.565884 (+ 1821us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:53.565910 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:53.566048 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:53.566405 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:53.566408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:53.566639 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:53.566825 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.566835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.569095 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.569099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.569917 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.569927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.570424 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.570428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.570531 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.571027 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:53.571038 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:53.571336 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:53.573350 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.573354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.573356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.573582 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.573893 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.573896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.573898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.573989 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.574381 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:53.574384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:53.574501 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 14:57:53.574629 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:57:53.574772 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"mutex_wait_us":26,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:57:53.586262 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37871
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:53.692690 9312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:53.692972 9312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:53.693020 9312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:53.697000 9312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:53.697049 9312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:53.697088 9312 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:57:53.697101 9312 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:57:53.697132 9312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:57:53.702188 9312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:37871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:53.703094 9312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:53.704028 9312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:53.712981 9320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:53.712988 9317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:53.713001 9318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:53.713140 9312 server_base.cc:1034] running on GCE node
I20260620 14:57:53.713650 9312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:53.714241 9312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:53.715432 9312 hybrid_clock.cc:648] HybridClock initialized: now 1781967473715416 us; error 33 us; skew 500 ppm
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:53.718199 9312 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:57:53.718984 9312 webserver.cc:460] Webserver started at http://127.7.46.130:37581/ using document root <none> and password file <none>
I20260620 14:57:53.719516 9312 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:53.719604 9312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:53.719884 9312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:53.721786 9312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "2916d65f7bc94872bdb053b937588e81"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "ef74f3afc2e5e1b480bf5c53c1133136"
server_key_iv: "5b2511d744cf6cbaa58a1771e2a6aa65"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.722333 9312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "2916d65f7bc94872bdb053b937588e81"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "ef74f3afc2e5e1b480bf5c53c1133136"
server_key_iv: "5b2511d744cf6cbaa58a1771e2a6aa65"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.725811 9312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:57:53.728171 9327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:53.729306 9312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:57:53.729436 9312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "2916d65f7bc94872bdb053b937588e81"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "ef74f3afc2e5e1b480bf5c53c1133136"
server_key_iv: "5b2511d744cf6cbaa58a1771e2a6aa65"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.729544 9312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:53.738133 9312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:53.740979 9312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:53.741173 9312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:53.742509 9312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:53.742581 9312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:53.742651 9312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:53.742703 9312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:53.753598 9312 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:38333
I20260620 14:57:53.753660 9442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:38333 every 8 connection(s)
I20260620 14:57:53.754472 9312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:57:53.756461 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.743306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:33791 (local address 127.7.46.190:37871)
0620 14:57:53.743449 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:53.743454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:53.744381 (+ 927us) server_negotiation.cc:407] Connection header received
0620 14:57:53.745647 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:53.745652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:53.745714 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:53.745816 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:53.747421 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.748213 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.749322 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.749593 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.749691 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.752828 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:53.752832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:53.752835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:53.752863 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:53.754438 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.754959 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.754963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.754965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.755085 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.755460 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.755465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.755467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.755652 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:53.755729 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:53.756251 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 14:57:53.756346 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:57:53.757267 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.743688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37871 (local address 127.7.46.130:33791)
0620 14:57:53.744208 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:53.744267 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:53.745388 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:53.746094 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:53.746105 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:53.746602 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:53.747277 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.747302 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.748332 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.748337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.749189 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.749198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.749758 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.749764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.750010 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.750650 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:53.750670 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:53.752651 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:53.754572 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.754577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.754590 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.754851 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.755185 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.755191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.755193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.755346 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.755750 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:53.755755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:53.756033 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 14:57:53.756197 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:57:53.756425 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":101,"threads_started":1}
I20260620 14:57:53.758495 9443 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37871
I20260620 14:57:53.758647 9443 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:53.759099 9443 heartbeater.cc:511] Master 127.7.46.190:37871 requested a full tablet report, sending...
I20260620 14:57:53.760131 9099 ts_manager.cc:194] Registered new tserver with Master: 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333)
I20260620 14:57:53.760777 9099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:33791
I20260620 14:57:53.761744 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 9312
I20260620 14:57:53.761857 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:57:53.762142 7354 external_mini_cluster.cc:1428] Setting key c55ed985e8cfcb9eaa957679eb391b1c
I20260620 14:57:53.768050 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.760503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:50841 (local address 127.7.46.190:37871)
0620 14:57:53.760627 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:53.760631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:53.760830 (+ 199us) server_negotiation.cc:407] Connection header received
0620 14:57:53.760946 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:53.760949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:53.760996 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:53.761066 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:53.761465 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.762215 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.763201 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.763470 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.763557 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.764720 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:53.764723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:53.764725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:53.764745 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:53.766381 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.766854 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.766856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.766857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.766946 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.767245 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.767247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.767248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.767431 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:53.767507 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:53.767850 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 14:57:53.767947 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:57:53.768181 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.760450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37871 (local address 127.7.46.130:50841)
0620 14:57:53.760754 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:53.760765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:53.760868 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:53.761079 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:53.761082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:53.761272 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:53.761357 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.761362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.762358 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.762362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.763059 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.763068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.763583 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.763587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.763691 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.764143 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:53.764163 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:53.764585 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:53.766500 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.766507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.766511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.766769 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.767036 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.767042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.767047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.767159 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.767532 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:53.767538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:53.767647 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 14:57:53.767796 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 14:57:53.767958 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:57:53.786731 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37871
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:53.894353 9452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:53.894640 9452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:53.894726 9452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:53.898679 9452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:53.898777 9452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:53.898849 9452 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:57:53.898892 9452 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:57:53.898959 9452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:57:53.904072 9452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:37871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:53.905120 9452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:53.906095 9452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:57:53.915316 9452 server_base.cc:1034] running on GCE node
W20260620 14:57:53.915280 9457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:53.915262 9458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:53.915302 9460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:53.915899 9452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:53.916541 9452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:53.917723 9452 hybrid_clock.cc:648] HybridClock initialized: now 1781967473917714 us; error 29 us; skew 500 ppm
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:53.920428 9452 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:57:53.921233 9452 webserver.cc:460] Webserver started at http://127.7.46.131:42471/ using document root <none> and password file <none>
I20260620 14:57:53.921823 9452 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:53.921912 9452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:53.922151 9452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:53.924149 9452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "f29d5723317b4f67be03e3573835d1cd"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "1c92296bad46ed4441fac80f8185682a"
server_key_iv: "962ea07bdf318456a782d3c3b89b251e"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.924693 9452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "f29d5723317b4f67be03e3573835d1cd"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "1c92296bad46ed4441fac80f8185682a"
server_key_iv: "962ea07bdf318456a782d3c3b89b251e"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.928274 9452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:57:53.930495 9467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:53.931738 9452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:53.931891 9452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "f29d5723317b4f67be03e3573835d1cd"
format_stamp: "Formatted at 2026-06-20 14:57:53 on dist-test-slave-30t4"
server_key: "1c92296bad46ed4441fac80f8185682a"
server_key_iv: "962ea07bdf318456a782d3c3b89b251e"
server_key_version: "encryptionkey@0"
I20260620 14:57:53.931998 9452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:53.948169 9452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:53.951258 9452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:53.951449 9452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:53.952832 9452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:53.952904 9452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:53.952973 9452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:53.953016 9452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:53.963227 9452 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:32879
I20260620 14:57:53.963299 9582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:32879 every 8 connection(s)
I20260620 14:57:53.964325 9452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:57:53.966666 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.953643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:41439 (local address 127.7.46.190:37871)
0620 14:57:53.953789 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:53.953793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:53.954610 (+ 817us) server_negotiation.cc:407] Connection header received
0620 14:57:53.955691 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:53.955694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:53.955748 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:53.955862 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:53.957389 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.958242 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.960165 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.960496 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.960646 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.963158 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:53.963162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:53.963164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:53.963191 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:53.964670 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.965233 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.965237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.965239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.965319 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.965653 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.965658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.965660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.965832 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:53.965932 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:53.966385 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 14:57:53.966497 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:57:53.967383 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.954017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37871 (local address 127.7.46.131:41439)
0620 14:57:53.954445 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:53.954490 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:53.955480 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:53.956075 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:53.956086 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:53.956561 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:53.957220 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.957242 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.958380 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.958385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.960009 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.960024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.960681 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.960683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.960819 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.961429 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:53.961465 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:53.962994 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:53.964779 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.964785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.964798 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.965134 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.965408 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.965412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.965414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.965552 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.965940 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:53.965947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:53.966167 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 14:57:53.966343 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 14:57:53.966632 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":123,"threads_started":1}
I20260620 14:57:53.968495 9583 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37871
I20260620 14:57:53.968665 9583 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:53.969115 9583 heartbeater.cc:511] Master 127.7.46.190:37871 requested a full tablet report, sending...
I20260620 14:57:53.970146 9100 ts_manager.cc:194] Registered new tserver with Master: f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:53.970892 9100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:41439
I20260620 14:57:53.972333 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 9452
I20260620 14:57:53.972445 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:57:53.972757 7354 external_mini_cluster.cc:1428] Setting key 36b80341876cc76e6bd0e225abaf4200
I20260620 14:57:53.974895 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:57:53.977813 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.970633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37871 (local address 127.7.46.131:55193)
0620 14:57:53.970942 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:53.970953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:53.971074 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:53.971310 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:53.971313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:53.971512 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:53.971626 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.971633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.972504 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.972508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.973217 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:53.973226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.973677 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.973680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:53.973754 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.974265 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:53.974279 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:53.974704 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:53.976369 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.976375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.976381 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.976645 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.976958 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:53.976960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:53.976962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:53.977035 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:53.977439 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:53.977441 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:53.977509 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 14:57:53.977586 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:57:53.977682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":97,"threads_started":1}
I20260620 14:57:53.978009 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.970738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:55193 (local address 127.7.46.190:37871)
0620 14:57:53.970852 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:53.970856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:53.971024 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:57:53.971165 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:53.971168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:53.971208 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:53.971297 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:53.971724 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.972347 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.973334 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.973559 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.973676 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.974828 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:53.974833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:53.974837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:53.974861 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:53.976266 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.976756 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.976761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.976765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.976853 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.977123 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.977128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.977130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.977344 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:53.977448 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:53.977675 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:57:53.977790 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:57:53 dist-test-slave-30t4 krb5kdc[9055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:54.000069 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:53.992093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.7.46.190:37871)
0620 14:57:53.992232 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:53.992235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:53.992247 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:57:53.992285 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:53.992287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:53.992324 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:53.992415 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:53.993152 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.993666 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.994540 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:53.994770 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:53.994879 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:53.997190 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:53.997193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:53.997195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:53.997221 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:53.998596 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.998978 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.998980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.998982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.999055 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:53.999276 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:53.999279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:53.999280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:53.999435 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:53.999547 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:53.999853 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 14:57:53.999945 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:57:54.002767 9100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:57:54.005692 9100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:54.019223 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.013499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42985 (local address 127.0.0.1:35570)
0620 14:57:54.013866 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.013910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.014025 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.014854 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.014860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.014882 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:54.015030 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.015039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.016302 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.016305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.017502 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.017513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.018658 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.018662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.018773 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.018818 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.018943 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.019073 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
I20260620 14:57:54.019236 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.013636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.7.46.129:42985)
0620 14:57:54.014305 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.014322 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.014352 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:57:54.014433 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.014438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.014681 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.014867 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:54.015178 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.016197 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.017651 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.018528 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.018655 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.018827 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.018995 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.019102 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":80,"threads_started":1}
I20260620 14:57:54.020002 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.014206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.7.46.130:38333)
0620 14:57:54.014606 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.014634 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.014903 (+ 269us) server_negotiation.cc:407] Connection header received
0620 14:57:54.015237 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.015245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.015437 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.015531 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:54.016120 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.016884 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.018395 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.019180 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.019285 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.019431 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.019745 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.019865 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":104,"threads_started":1}
I20260620 14:57:54.020437 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.013833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:38333 (local address 127.0.0.1:52326)
0620 14:57:54.014806 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.014821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.015062 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.015695 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.015698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.015731 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:54.015994 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.016002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.017011 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.017015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.018263 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.018274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.019488 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.019492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.019592 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.019613 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.020219 (+ 606us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.020291 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":111,"spinlock_wait_cycles":6272,"thread_start_us":103,"threads_started":1}
I20260620 14:57:54.020864 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.014220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:32879 (local address 127.0.0.1:45314)
0620 14:57:54.015137 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.015153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.015229 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.015695 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.015698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.015732 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:54.016009 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.016015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.018085 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.018089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.019167 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.019178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.020459 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.020464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.020571 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.020607 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.020673 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.020743 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":78,"spinlock_wait_cycles":4992,"thread_start_us":87,"threads_started":1}
I20260620 14:57:54.021250 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.014366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.7.46.131:32879)
0620 14:57:54.015252 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.015264 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.015287 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:57:54.015354 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.015359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.015542 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.015696 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:54.017002 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.017970 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.019306 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.020129 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.020234 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.020399 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.021007 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.021110 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":88,"threads_started":1}
I20260620 14:57:54.022580 9237 tablet_service.cc:1467] Processing CreateTablet for tablet 255e4ac8e8814d339995ac64bbb7c1e1 (DEFAULT_TABLE table=test-table [id=ebf17b6bc12143008c5cd57c26bb6cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:54.022756 9377 tablet_service.cc:1467] Processing CreateTablet for tablet 255e4ac8e8814d339995ac64bbb7c1e1 (DEFAULT_TABLE table=test-table [id=ebf17b6bc12143008c5cd57c26bb6cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:54.023597 9377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255e4ac8e8814d339995ac64bbb7c1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.023653 9237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255e4ac8e8814d339995ac64bbb7c1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.025547 9517 tablet_service.cc:1467] Processing CreateTablet for tablet 255e4ac8e8814d339995ac64bbb7c1e1 (DEFAULT_TABLE table=test-table [id=ebf17b6bc12143008c5cd57c26bb6cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:54.026501 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255e4ac8e8814d339995ac64bbb7c1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.030676 9606 tablet_bootstrap.cc:492] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Bootstrap starting.
I20260620 14:57:54.032049 9607 tablet_bootstrap.cc:492] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Bootstrap starting.
I20260620 14:57:54.033039 9606 tablet_bootstrap.cc:654] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.033733 9608 tablet_bootstrap.cc:492] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Bootstrap starting.
I20260620 14:57:54.033977 9606 log.cc:824] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:54.034281 9607 tablet_bootstrap.cc:654] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.035140 9607 log.cc:824] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:54.036271 9608 tablet_bootstrap.cc:654] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.036859 9607 tablet_bootstrap.cc:492] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: No bootstrap required, opened a new log
I20260620 14:57:54.036938 9606 tablet_bootstrap.cc:492] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: No bootstrap required, opened a new log
I20260620 14:57:54.037030 9607 ts_tablet_manager.cc:1397] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 14:57:54.037133 9606 ts_tablet_manager.cc:1397] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:57:54.037218 9608 log.cc:824] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:54.039152 9608 tablet_bootstrap.cc:492] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: No bootstrap required, opened a new log
I20260620 14:57:54.039335 9608 ts_tablet_manager.cc:1397] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:57:54.040076 9607 raft_consensus.cc:348] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.040295 9607 raft_consensus.cc:374] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.040345 9607 raft_consensus.cc:729] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2916d65f7bc94872bdb053b937588e81, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.040766 9607 consensus_queue.cc:260] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.041379 9443 heartbeater.cc:503] Master 127.7.46.190:37871 was elected leader, sending a full tablet report...
I20260620 14:57:54.041294 9606 raft_consensus.cc:348] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.041533 9606 raft_consensus.cc:374] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.041584 9606 raft_consensus.cc:729] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.041707 9607 ts_tablet_manager.cc:1428] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 14:57:54.042344 9606 consensus_queue.cc:260] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.043134 9303 heartbeater.cc:503] Master 127.7.46.190:37871 was elected leader, sending a full tablet report...
I20260620 14:57:54.043514 9606 ts_tablet_manager.cc:1428] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:57:54.043395 9608 raft_consensus.cc:348] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.043633 9608 raft_consensus.cc:374] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.043682 9608 raft_consensus.cc:729] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29d5723317b4f67be03e3573835d1cd, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.044301 9608 consensus_queue.cc:260] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.044996 9583 heartbeater.cc:503] Master 127.7.46.190:37871 was elected leader, sending a full tablet report...
I20260620 14:57:54.045298 9608 ts_tablet_manager.cc:1428] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260620 14:57:54.059024 9304 tablet.cc:2369] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:54.215798 9584 tablet.cc:2369] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:54.255872 9444 tablet.cc:2369] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:57:54.300679 9613 raft_consensus.cc:482] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:54.300908 9613 raft_consensus.cc:504] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.302018 9613 leader_election.cc:288] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879), 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333)
I20260620 14:57:54.305797 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.302409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:40989 (local address 127.7.46.130:38333)
0620 14:57:54.302520 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.302524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.302779 (+ 255us) server_negotiation.cc:407] Connection header received
0620 14:57:54.302894 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.302897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.302947 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.303034 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:54.303305 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.303833 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.304835 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.305257 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.305381 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.305436 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.305594 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.305673 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:57:54.305802 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.302332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:38333 (local address 127.7.46.129:40989)
0620 14:57:54.302671 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.302686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.302782 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.303052 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.303054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.303068 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:54.303192 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.303200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.303939 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.303942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.304717 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.304725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.305377 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.305381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.305473 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.305499 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.305557 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.305641 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260620 14:57:54.305835 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.302344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:39163 (local address 127.7.46.131:32879)
0620 14:57:54.302485 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.302490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.302649 (+ 159us) server_negotiation.cc:407] Connection header received
0620 14:57:54.302748 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.302751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.302798 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.302875 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:54.303150 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.303722 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.304878 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.305262 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.305387 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.305435 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.305660 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.305729 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:57:54.305824 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.302187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:32879 (local address 127.7.46.129:39163)
0620 14:57:54.302546 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.302561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.302661 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.302897 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.302900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.302917 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:54.303046 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.303052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.303857 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.303863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.304731 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.304740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.305397 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.305400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.305517 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.305534 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.305629 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.305692 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":20,"thread_start_us":82,"threads_started":1}
I20260620 14:57:54.306440 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255e4ac8e8814d339995ac64bbb7c1e1" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2916d65f7bc94872bdb053b937588e81" is_pre_election: true
I20260620 14:57:54.306604 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255e4ac8e8814d339995ac64bbb7c1e1" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd" is_pre_election: true
I20260620 14:57:54.306725 9397 raft_consensus.cc:2454] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 0.
I20260620 14:57:54.306888 9537 raft_consensus.cc:2454] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 0.
I20260620 14:57:54.307228 9191 leader_election.cc:302] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2916d65f7bc94872bdb053b937588e81, efbe10cd8969432bbf558b81bd76c46c; no voters:
I20260620 14:57:54.307482 9613 raft_consensus.cc:2781] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:54.307554 9613 raft_consensus.cc:482] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:54.307610 9613 raft_consensus.cc:3037] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.308629 9613 raft_consensus.cc:504] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.308946 9613 leader_election.cc:288] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 election: Requested vote from peers f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879), 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333)
I20260620 14:57:54.309255 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255e4ac8e8814d339995ac64bbb7c1e1" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd"
I20260620 14:57:54.309374 9537 raft_consensus.cc:3037] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.309448 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255e4ac8e8814d339995ac64bbb7c1e1" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2916d65f7bc94872bdb053b937588e81"
I20260620 14:57:54.309597 9397 raft_consensus.cc:3037] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.310552 9397 raft_consensus.cc:2454] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 1.
I20260620 14:57:54.310834 9191 leader_election.cc:302] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2916d65f7bc94872bdb053b937588e81, efbe10cd8969432bbf558b81bd76c46c; no voters:
I20260620 14:57:54.310943 9537 raft_consensus.cc:2454] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 1.
I20260620 14:57:54.311028 9613 raft_consensus.cc:2781] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:54.311189 9613 raft_consensus.cc:686] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 1 LEADER]: Becoming Leader. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Running, Role: LEADER
I20260620 14:57:54.311545 9613 consensus_queue.cc:237] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } }
I20260620 14:57:54.314535 9098 catalog_manager.cc:5495] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c reported cstate change: term changed from 0 to 1, leader changed from <none> to efbe10cd8969432bbf558b81bd76c46c (127.7.46.129). New cstate: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:57:54 dist-test-slave-30t4 krb5kdc[9055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967473, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:54.401278 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.393436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37871 (local address 127.0.0.1:55912)
0620 14:57:54.393736 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.393748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.393865 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.394203 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.394205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.394565 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:54.394673 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.394679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.395392 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.395395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.395958 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.395966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.396367 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.396370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.396429 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.396938 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:54.396949 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:54.398271 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:54.400018 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:54.400025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:54.400029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:54.400242 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:54.400462 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:54.400464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:54.400465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:54.400534 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:54.400835 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:54.400839 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:54.400927 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.401013 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.401130 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260620 14:57:54.401297 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.393475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.7.46.190:37871)
0620 14:57:54.393588 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.393611 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.393820 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:57:54.394041 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.394044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.394095 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.394171 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:54.394796 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.395298 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.396061 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.396264 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.396349 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.398388 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:54.398392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:54.398394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:54.398414 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:54.399933 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:54.400321 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:54.400324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:54.400325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:54.400390 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:54.400604 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:54.400606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:54.400607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:54.400753 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:54.400826 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.401065 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.401181 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":42}
I20260620 14:57:54.404942 9098 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:55912:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:57:54.405591 9098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:54.410243 9377 tablet_service.cc:1467] Processing CreateTablet for tablet 00df5573a5cf4f2c8741d396292ab252 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3593f982588e43ee91860716df23750d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:57:54.410202 9237 tablet_service.cc:1467] Processing CreateTablet for tablet 00df5573a5cf4f2c8741d396292ab252 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3593f982588e43ee91860716df23750d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:57:54.410310 9517 tablet_service.cc:1467] Processing CreateTablet for tablet 00df5573a5cf4f2c8741d396292ab252 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3593f982588e43ee91860716df23750d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:57:54.410866 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00df5573a5cf4f2c8741d396292ab252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.410866 9237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00df5573a5cf4f2c8741d396292ab252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.410866 9377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00df5573a5cf4f2c8741d396292ab252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.413595 9607 tablet_bootstrap.cc:492] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: Bootstrap starting.
I20260620 14:57:54.413618 9608 tablet_bootstrap.cc:492] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: Bootstrap starting.
I20260620 14:57:54.413815 9606 tablet_bootstrap.cc:492] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: Bootstrap starting.
I20260620 14:57:54.414564 9607 tablet_bootstrap.cc:654] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.414649 9608 tablet_bootstrap.cc:654] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.415140 9606 tablet_bootstrap.cc:654] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.415628 9607 tablet_bootstrap.cc:492] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: No bootstrap required, opened a new log
I20260620 14:57:54.415716 9607 ts_tablet_manager.cc:1397] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 14:57:54.416251 9607 raft_consensus.cc:348] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.416342 9606 tablet_bootstrap.cc:492] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: No bootstrap required, opened a new log
I20260620 14:57:54.416356 9607 raft_consensus.cc:374] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.416389 9607 raft_consensus.cc:729] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2916d65f7bc94872bdb053b937588e81, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.416417 9606 ts_tablet_manager.cc:1397] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:54.416461 9608 tablet_bootstrap.cc:492] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: No bootstrap required, opened a new log
I20260620 14:57:54.416518 9608 ts_tablet_manager.cc:1397] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:54.416513 9607 consensus_queue.cc:260] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.416772 9607 ts_tablet_manager.cc:1428] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:54.416841 9608 raft_consensus.cc:348] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.416911 9608 raft_consensus.cc:374] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.416934 9608 raft_consensus.cc:729] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29d5723317b4f67be03e3573835d1cd, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.416882 9606 raft_consensus.cc:348] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.416993 9606 raft_consensus.cc:374] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.417029 9606 raft_consensus.cc:729] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.417059 9608 consensus_queue.cc:260] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.417153 9606 consensus_queue.cc:260] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.417323 9608 ts_tablet_manager.cc:1428] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:57:54.417304 9614 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.417419 9606 ts_tablet_manager.cc:1428] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:57:54.417446 9614 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:54.417600 9613 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.417714 9613 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:54.418210 9612 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.418334 9612 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:54.589756 9613 raft_consensus.cc:482] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:54.589898 9613 raft_consensus.cc:504] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.590399 9613 leader_election.cc:288] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333), f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:54.590904 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00df5573a5cf4f2c8741d396292ab252" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd" is_pre_election: true
I20260620 14:57:54.590862 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00df5573a5cf4f2c8741d396292ab252" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2916d65f7bc94872bdb053b937588e81" is_pre_election: true
I20260620 14:57:54.591053 9537 raft_consensus.cc:2454] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 0.
I20260620 14:57:54.591120 9397 raft_consensus.cc:2454] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 0.
I20260620 14:57:54.591341 9189 leader_election.cc:302] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efbe10cd8969432bbf558b81bd76c46c, f29d5723317b4f67be03e3573835d1cd; no voters:
I20260620 14:57:54.591545 9613 raft_consensus.cc:2781] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:54.591612 9613 raft_consensus.cc:482] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:54.591660 9613 raft_consensus.cc:3037] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.592796 9613 raft_consensus.cc:504] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.593108 9613 leader_election.cc:288] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 election: Requested vote from peers 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333), f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:54.593449 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00df5573a5cf4f2c8741d396292ab252" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2916d65f7bc94872bdb053b937588e81"
I20260620 14:57:54.593536 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00df5573a5cf4f2c8741d396292ab252" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd"
I20260620 14:57:54.593609 9397 raft_consensus.cc:3037] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.593673 9537 raft_consensus.cc:3037] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.594493 9397 raft_consensus.cc:2454] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 1.
I20260620 14:57:54.594517 9537 raft_consensus.cc:2454] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 1.
I20260620 14:57:54.594787 9191 leader_election.cc:302] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2916d65f7bc94872bdb053b937588e81, efbe10cd8969432bbf558b81bd76c46c; no voters:
I20260620 14:57:54.595011 9613 raft_consensus.cc:2781] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:54.595117 9613 raft_consensus.cc:686] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [term 1 LEADER]: Becoming Leader. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Running, Role: LEADER
I20260620 14:57:54.595242 9613 consensus_queue.cc:237] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.595862 9617 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: TxnStatusTablet state changed. Reason: New leader efbe10cd8969432bbf558b81bd76c46c. Latest consensus state: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.595960 9617 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:57:54.596311 9629 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:57:54.596550 9629 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:57:54.597019 9098 catalog_manager.cc:5495] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c reported cstate change: term changed from 0 to 1, leader changed from <none> to efbe10cd8969432bbf558b81bd76c46c (127.7.46.129). New cstate: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } health_report { overall_health: UNKNOWN } } }
I20260620 14:57:54.608951 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.604551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42985 (local address 127.0.0.1:35580)
0620 14:57:54.604727 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.604762 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.604844 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.605862 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.605865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.605874 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:57:54.605984 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.605989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.606757 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.606762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.607359 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.607366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.607913 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.607917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.608013 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.608055 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.608673 (+ 618us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.608687 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.608757 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.608836 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:57:54.608978 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.605344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.7.46.129:42985)
0620 14:57:54.605589 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.605595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.605606 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:57:54.605665 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.605667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.605713 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.605849 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:54.606105 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.606648 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.607482 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.607753 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.607915 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.608197 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.608578 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.608695 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.608766 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.608814 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260620 14:57:54.613803 9397 raft_consensus.cc:1261] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Refusing update from remote peer efbe10cd8969432bbf558b81bd76c46c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:54.613889 9537 raft_consensus.cc:1261] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Refusing update from remote peer efbe10cd8969432bbf558b81bd76c46c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:54.614457 9617 consensus_queue.cc:1035] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:54.614607 9613 consensus_queue.cc:1035] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:54.618813 9614 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: TxnStatusTablet state changed. Reason: New leader efbe10cd8969432bbf558b81bd76c46c. Latest consensus state: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.618933 9614 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:54.619223 9612 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: TxnStatusTablet state changed. Reason: New leader efbe10cd8969432bbf558b81bd76c46c. Latest consensus state: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.619325 9612 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:54.625494 9633 mvcc.cc:205] Tried to move back new op lower bound from 7298938776011325440 to 7298938775943340032. Current Snapshot: MvccSnapshot[applied={T|T < 7298938776011325440}]
I20260620 14:57:54.625608 9613 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.625713 9613 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:57:54.625933 9614 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.626036 9614 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P f29d5723317b4f67be03e3573835d1cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:54.627739 9617 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.627861 9617 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P efbe10cd8969432bbf558b81bd76c46c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:57:54.628991 9612 tablet_replica.cc:439] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } } }
I20260620 14:57:54.629109 9612 tablet_replica.cc:442] T 00df5573a5cf4f2c8741d396292ab252 P 2916d65f7bc94872bdb053b937588e81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:57:54.629698 9634 mvcc.cc:205] Tried to move back new op lower bound from 7298938776011325440 to 7298938775943340032. Current Snapshot: MvccSnapshot[applied={T|T < 7298938776011325440}]
I20260620 14:57:54.636873 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.633123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.7.46.129:42985)
0620 14:57:54.633232 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.633237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.633438 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:57:54.633612 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.633615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.633673 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.633755 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:54.633967 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.634770 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.635663 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.636046 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.636165 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.636428 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.636512 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.636585 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.636697 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.636747 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:57:54.647094 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.643561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42985 (local address 127.7.46.129:39331)
0620 14:57:54.643894 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.643906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.643985 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.644197 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.644199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.644205 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:57:54.644331 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.644335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.645257 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.645259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.645889 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.645896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.646436 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.646438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.646507 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.646544 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.646838 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.646843 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.646890 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.646961 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":20,"thread_start_us":101,"threads_started":1}
I20260620 14:57:54.647107 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.643627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:39331 (local address 127.7.46.129:42985)
0620 14:57:54.643740 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.643744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.643976 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:57:54.644058 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.644061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.644106 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.644180 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:54.644427 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.645161 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.645993 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.646332 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.646430 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.646688 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.646759 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.646828 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.646900 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.646973 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":51}
I20260620 14:57:54.653182 9397 raft_consensus.cc:1261] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Refusing update from remote peer efbe10cd8969432bbf558b81bd76c46c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:54.653208 9537 raft_consensus.cc:1261] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Refusing update from remote peer efbe10cd8969432bbf558b81bd76c46c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:54.653569 9613 consensus_queue.cc:1035] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:54.653771 9617 consensus_queue.cc:1035] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:54.691330 9098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:54.691522 9098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:54.693905 9098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: Sending DeleteTablet for 3 replicas of tablet 255e4ac8e8814d339995ac64bbb7c1e1
I20260620 14:57:54.694605 9377 tablet_service.cc:1514] Processing DeleteTablet for tablet 255e4ac8e8814d339995ac64bbb7c1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:54 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:52326
I20260620 14:57:54.694769 9517 tablet_service.cc:1514] Processing DeleteTablet for tablet 255e4ac8e8814d339995ac64bbb7c1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:54 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:45314
I20260620 14:57:54.695091 9098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:57:54.695199 9236 tablet_service.cc:1514] Processing DeleteTablet for tablet 255e4ac8e8814d339995ac64bbb7c1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:54 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:35570
W20260620 14:57:54.695473 9098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:54.695621 9653 tablet_replica.cc:330] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: stopping tablet replica
I20260620 14:57:54.695636 9651 tablet_replica.cc:330] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: stopping tablet replica
I20260620 14:57:54.695977 9651 raft_consensus.cc:2229] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:54.695981 9653 raft_consensus.cc:2229] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:54.696391 9653 raft_consensus.cc:2258] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:54.696604 9651 raft_consensus.cc:2258] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:54.697649 9653 ts_tablet_manager.cc:1905] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:54.697808 9651 ts_tablet_manager.cc:1905] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:54.699944 9377 tablet_service.cc:1467] Processing CreateTablet for tablet 28a14858c0934eb9bbf3f145fc618017 (DEFAULT_TABLE table=test-table [id=7815159ec81a4d4c84d1fd182fcd7510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:54.700157 9236 tablet_service.cc:1467] Processing CreateTablet for tablet 28a14858c0934eb9bbf3f145fc618017 (DEFAULT_TABLE table=test-table [id=7815159ec81a4d4c84d1fd182fcd7510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:54.700205 9377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a14858c0934eb9bbf3f145fc618017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.700361 9236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a14858c0934eb9bbf3f145fc618017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.701951 9607 tablet_bootstrap.cc:492] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: Bootstrap starting.
I20260620 14:57:54.702571 9517 tablet_service.cc:1467] Processing CreateTablet for tablet 28a14858c0934eb9bbf3f145fc618017 (DEFAULT_TABLE table=test-table [id=7815159ec81a4d4c84d1fd182fcd7510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:54.702795 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a14858c0934eb9bbf3f145fc618017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:54.702872 9607 tablet_bootstrap.cc:654] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.703298 9651 ts_tablet_manager.cc:1918] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:54.703377 9651 log.cc:1196] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/255e4ac8e8814d339995ac64bbb7c1e1
I20260620 14:57:54.703703 9651 ts_tablet_manager.cc:1939] T 255e4ac8e8814d339995ac64bbb7c1e1 P 2916d65f7bc94872bdb053b937588e81: Deleting consensus metadata
I20260620 14:57:54.703923 9652 tablet_replica.cc:330] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: stopping tablet replica
I20260620 14:57:54.704212 9652 raft_consensus.cc:2229] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:54.704416 9652 raft_consensus.cc:2258] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:54.704774 9607 tablet_bootstrap.cc:492] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: No bootstrap required, opened a new log
I20260620 14:57:54.704838 9607 ts_tablet_manager.cc:1397] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:54.705067 9606 tablet_bootstrap.cc:492] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: Bootstrap starting.
I20260620 14:57:54.705220 9607 raft_consensus.cc:348] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.705317 9607 raft_consensus.cc:374] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.705348 9607 raft_consensus.cc:729] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2916d65f7bc94872bdb053b937588e81, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.705466 9607 consensus_queue.cc:260] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.705739 9607 ts_tablet_manager.cc:1428] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:57:54.705922 9606 tablet_bootstrap.cc:654] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.706476 9608 tablet_bootstrap.cc:492] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: Bootstrap starting.
I20260620 14:57:54.706504 9652 ts_tablet_manager.cc:1905] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:54.707305 9608 tablet_bootstrap.cc:654] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:54.709185 9652 ts_tablet_manager.cc:1918] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:54.709282 9652 log.cc:1196] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/255e4ac8e8814d339995ac64bbb7c1e1
I20260620 14:57:54.709576 9653 ts_tablet_manager.cc:1918] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:54.709617 9652 ts_tablet_manager.cc:1939] T 255e4ac8e8814d339995ac64bbb7c1e1 P f29d5723317b4f67be03e3573835d1cd: Deleting consensus metadata
I20260620 14:57:54.709719 9653 log.cc:1196] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/255e4ac8e8814d339995ac64bbb7c1e1
I20260620 14:57:54.709987 9608 tablet_bootstrap.cc:492] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: No bootstrap required, opened a new log
I20260620 14:57:54.710067 9608 ts_tablet_manager.cc:1397] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:57:54.710193 9653 ts_tablet_manager.cc:1939] T 255e4ac8e8814d339995ac64bbb7c1e1 P efbe10cd8969432bbf558b81bd76c46c: Deleting consensus metadata
I20260620 14:57:54.710460 9608 raft_consensus.cc:348] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.710569 9608 raft_consensus.cc:374] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.710605 9608 raft_consensus.cc:729] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29d5723317b4f67be03e3573835d1cd, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.710716 9608 consensus_queue.cc:260] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.710734 9085 catalog_manager.cc:4841] TS f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879): tablet 255e4ac8e8814d339995ac64bbb7c1e1 (table test-table [id=ebf17b6bc12143008c5cd57c26bb6cc5]) successfully deleted
I20260620 14:57:54.710901 9087 catalog_manager.cc:4841] TS 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333): tablet 255e4ac8e8814d339995ac64bbb7c1e1 (table test-table [id=ebf17b6bc12143008c5cd57c26bb6cc5]) successfully deleted
I20260620 14:57:54.710984 9608 ts_tablet_manager.cc:1428] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:54.711700 9606 tablet_bootstrap.cc:492] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: No bootstrap required, opened a new log
I20260620 14:57:54.711808 9606 ts_tablet_manager.cc:1397] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 14:57:54.712191 9606 raft_consensus.cc:348] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.712294 9606 raft_consensus.cc:374] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:54.712333 9606 raft_consensus.cc:729] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Initialized, Role: FOLLOWER
I20260620 14:57:54.712445 9606 consensus_queue.cc:260] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.712680 9606 ts_tablet_manager.cc:1428] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:54.714442 9085 catalog_manager.cc:4841] TS efbe10cd8969432bbf558b81bd76c46c (127.7.46.129:42985): tablet 255e4ac8e8814d339995ac64bbb7c1e1 (table test-table [id=ebf17b6bc12143008c5cd57c26bb6cc5]) successfully deleted
I20260620 14:57:54.745815 9612 raft_consensus.cc:482] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:54.745960 9612 raft_consensus.cc:504] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.747196 9612 leader_election.cc:288] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efbe10cd8969432bbf558b81bd76c46c (127.7.46.129:42985), f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:54.752976 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.748738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:56637 (local address 127.7.46.129:42985)
0620 14:57:54.748841 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.748845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.748858 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:57:54.748905 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.748908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.748947 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.749014 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:54.749369 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.750120 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.751659 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.752287 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.752385 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.752434 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.752784 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.752853 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:57:54.753360 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.747543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42985 (local address 127.7.46.130:56637)
0620 14:57:54.747920 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.747964 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.748136 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.749129 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.749132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.749148 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:54.749259 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.749264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.750254 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.750257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.751524 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.751534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.752536 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.752540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.752635 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.752678 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.753145 (+ 467us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.753216 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260620 14:57:54.753942 9257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a14858c0934eb9bbf3f145fc618017" candidate_uuid: "2916d65f7bc94872bdb053b937588e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbe10cd8969432bbf558b81bd76c46c" is_pre_election: true
I20260620 14:57:54.754205 9257 raft_consensus.cc:2454] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2916d65f7bc94872bdb053b937588e81 in term 0.
I20260620 14:57:54.754550 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.748597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:50671 (local address 127.7.46.131:32879)
0620 14:57:54.748826 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.748832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.748876 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:57:54.748960 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.748966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.749050 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.749171 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:54.749830 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.750802 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.752924 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.753851 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.753966 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.754013 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.754346 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.754427 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:57:54.754634 9329 leader_election.cc:302] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2916d65f7bc94872bdb053b937588e81, efbe10cd8969432bbf558b81bd76c46c; no voters:
I20260620 14:57:54.754878 9612 raft_consensus.cc:2781] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:54.754928 9612 raft_consensus.cc:482] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:54.754935 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.747920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:32879 (local address 127.7.46.130:50671)
0620 14:57:54.748329 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.748343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.748431 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.749355 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.749360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.749406 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:54.749606 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.749616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.750963 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.750966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.752721 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.752732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.754075 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.754090 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.754194 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.754221 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.754728 (+ 507us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.754796 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":22,"thread_start_us":92,"threads_started":1}
I20260620 14:57:54.755002 9612 raft_consensus.cc:3037] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.756522 9612 raft_consensus.cc:504] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.756768 9612 leader_election.cc:288] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [CANDIDATE]: Term 1 election: Requested vote from peers efbe10cd8969432bbf558b81bd76c46c (127.7.46.129:42985), f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:54.757498 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a14858c0934eb9bbf3f145fc618017" candidate_uuid: "2916d65f7bc94872bdb053b937588e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd" is_pre_election: true
I20260620 14:57:54.757535 9536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a14858c0934eb9bbf3f145fc618017" candidate_uuid: "2916d65f7bc94872bdb053b937588e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd"
I20260620 14:57:54.757632 9537 raft_consensus.cc:2454] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2916d65f7bc94872bdb053b937588e81 in term 0.
I20260620 14:57:54.758086 9257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a14858c0934eb9bbf3f145fc618017" candidate_uuid: "2916d65f7bc94872bdb053b937588e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbe10cd8969432bbf558b81bd76c46c"
I20260620 14:57:54.758193 9257 raft_consensus.cc:3037] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.759160 9257 raft_consensus.cc:2454] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2916d65f7bc94872bdb053b937588e81 in term 1.
I20260620 14:57:54.759492 9329 leader_election.cc:302] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2916d65f7bc94872bdb053b937588e81, efbe10cd8969432bbf558b81bd76c46c; no voters: f29d5723317b4f67be03e3573835d1cd
I20260620 14:57:54.759898 9612 raft_consensus.cc:2781] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:54.760082 9612 raft_consensus.cc:686] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 1 LEADER]: Becoming Leader. State: Replica: 2916d65f7bc94872bdb053b937588e81, State: Running, Role: LEADER
I20260620 14:57:54.760321 9612 consensus_queue.cc:237] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:54.762398 9099 catalog_manager.cc:5495] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2916d65f7bc94872bdb053b937588e81 (127.7.46.130). New cstate: current_term: 1 leader_uuid: "2916d65f7bc94872bdb053b937588e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } health_report { overall_health: UNKNOWN } } }
I20260620 14:57:54.786118 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.782654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.7.46.130:38333)
0620 14:57:54.782754 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.782759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.782841 (+ 82us) server_negotiation.cc:407] Connection header received
0620 14:57:54.783019 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.783022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.783077 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.783190 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:54.783410 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.783975 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.784778 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.784997 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.785109 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.785357 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.785747 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.785863 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.785934 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.785985 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:57:54.796038 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.792759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42985 (local address 127.7.46.130:59203)
0620 14:57:54.793001 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.793036 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.793138 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.793379 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.793381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.793388 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:57:54.793480 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.793485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.794269 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.794273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.794879 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.794887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.795320 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.795324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.795407 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.795451 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.795746 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.795752 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.795831 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.795906 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260620 14:57:54.796027 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.792838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:59203 (local address 127.7.46.129:42985)
0620 14:57:54.792958 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.792963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.793088 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:57:54.793234 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.793237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.793292 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.793370 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:54.793592 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.794163 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.794994 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.795214 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.795318 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.795575 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.795649 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.795722 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.795850 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.795895 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:57:54.802443 9537 raft_consensus.cc:3037] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:54.802737 9257 raft_consensus.cc:1261] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Refusing update from remote peer 2916d65f7bc94872bdb053b937588e81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:54.803216 9612 consensus_queue.cc:1035] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:54.803965 9537 raft_consensus.cc:1261] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Refusing update from remote peer 2916d65f7bc94872bdb053b937588e81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:54.804349 9612 consensus_queue.cc:1035] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:54.809528 9635 mvcc.cc:205] Tried to move back new op lower bound from 7298938776786972672 to 7298938776620789760. Current Snapshot: MvccSnapshot[applied={T|T < 7298938776786972672}]
I20260620 14:57:54.823130 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.820104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:38333 (local address 127.7.46.129:35527)
0620 14:57:54.820250 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:54.820266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:54.820342 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:54.820619 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:54.820622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:54.820629 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:57:54.820725 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.820730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.821444 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.821450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.822035 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:54.822051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.822480 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.822484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:54.822574 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.822598 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.822870 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.822876 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:57:54.822932 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:57:54.823012 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 14:57:54.823123 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:54.820323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:35527 (local address 127.7.46.130:38333)
0620 14:57:54.820422 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:54.820426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:54.820440 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:57:54.820481 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:54.820483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:54.820521 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:54.820582 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:54.820846 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.821337 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.822164 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:54.822376 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:54.822481 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:54.822717 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:54.822781 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:54.822854 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:54.822951 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:57:54.822996 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 14:57:54.824084 9634 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:57:56.381314 9100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:56.381453 9100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:56.383627 9100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: Sending DeleteTablet for 3 replicas of tablet 28a14858c0934eb9bbf3f145fc618017
I20260620 14:57:56.384186 9236 tablet_service.cc:1514] Processing DeleteTablet for tablet 28a14858c0934eb9bbf3f145fc618017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:56 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:35570
I20260620 14:57:56.384200 9377 tablet_service.cc:1514] Processing DeleteTablet for tablet 28a14858c0934eb9bbf3f145fc618017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:56 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:52326
I20260620 14:57:56.384233 9517 tablet_service.cc:1514] Processing DeleteTablet for tablet 28a14858c0934eb9bbf3f145fc618017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:56 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:45314
I20260620 14:57:56.384678 9690 tablet_replica.cc:330] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: stopping tablet replica
I20260620 14:57:56.384816 9690 raft_consensus.cc:2229] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:56.384919 9690 raft_consensus.cc:2258] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:56.385125 9692 tablet_replica.cc:330] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: stopping tablet replica
I20260620 14:57:56.385246 9692 raft_consensus.cc:2229] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:56.385370 9692 raft_consensus.cc:2258] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:56.385601 9691 tablet_replica.cc:330] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: stopping tablet replica
I20260620 14:57:56.385723 9691 raft_consensus.cc:2229] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:56.385977 9692 ts_tablet_manager.cc:1905] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:56.385975 9691 raft_consensus.cc:2258] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:56.386530 9691 ts_tablet_manager.cc:1905] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:56.387856 9690 ts_tablet_manager.cc:1905] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:56.388667 9692 ts_tablet_manager.cc:1918] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:57:56.388779 9692 log.cc:1196] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/28a14858c0934eb9bbf3f145fc618017
I20260620 14:57:56.389091 9691 ts_tablet_manager.cc:1918] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:57:56.389139 9692 ts_tablet_manager.cc:1939] T 28a14858c0934eb9bbf3f145fc618017 P f29d5723317b4f67be03e3573835d1cd: Deleting consensus metadata
I20260620 14:57:56.389199 9691 log.cc:1196] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/28a14858c0934eb9bbf3f145fc618017
I20260620 14:57:56.389472 9691 ts_tablet_manager.cc:1939] T 28a14858c0934eb9bbf3f145fc618017 P 2916d65f7bc94872bdb053b937588e81: Deleting consensus metadata
I20260620 14:57:56.389912 9085 catalog_manager.cc:4841] TS f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879): tablet 28a14858c0934eb9bbf3f145fc618017 (table test-table [id=7815159ec81a4d4c84d1fd182fcd7510]) successfully deleted
I20260620 14:57:56.390125 9087 catalog_manager.cc:4841] TS 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333): tablet 28a14858c0934eb9bbf3f145fc618017 (table test-table [id=7815159ec81a4d4c84d1fd182fcd7510]) successfully deleted
I20260620 14:57:56.390806 9690 ts_tablet_manager.cc:1918] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:57:56.390866 9690 log.cc:1196] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/28a14858c0934eb9bbf3f145fc618017
I20260620 14:57:56.391108 9690 ts_tablet_manager.cc:1939] T 28a14858c0934eb9bbf3f145fc618017 P efbe10cd8969432bbf558b81bd76c46c: Deleting consensus metadata
I20260620 14:57:56.391760 9085 catalog_manager.cc:4841] TS efbe10cd8969432bbf558b81bd76c46c (127.7.46.129:42985): tablet 28a14858c0934eb9bbf3f145fc618017 (table test-table [id=7815159ec81a4d4c84d1fd182fcd7510]) successfully deleted
Jun 20 14:57:56 dist-test-slave-30t4 krb5kdc[9055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:57:56 dist-test-slave-30t4 krb5kdc[9055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:56.410068 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:56.401972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.7.46.190:37871)
0620 14:57:56.402205 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:56.402211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:56.402280 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:57:56.402375 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:56.402377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:56.402416 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:56.402491 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:56.403112 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:56.403634 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:56.404457 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:56.404672 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:56.404747 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:56.406935 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:56.406938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:56.406940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:56.406961 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:56.408467 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:56.408885 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:56.408890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:56.408892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:56.408968 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:56.409235 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:56.409239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:56.409241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:56.409410 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:56.409548 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:56.409777 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:57:56.409896 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260620 14:57:56.412511 9100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:57:56.412932 9100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:56.417389 9377 tablet_service.cc:1467] Processing CreateTablet for tablet ea181a2a87634b5f9421f5cd13d32063 (DEFAULT_TABLE table=test-table [id=a6cd2e9f6c404ae2aeb37c43a73ad3b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:56.417368 9236 tablet_service.cc:1467] Processing CreateTablet for tablet ea181a2a87634b5f9421f5cd13d32063 (DEFAULT_TABLE table=test-table [id=a6cd2e9f6c404ae2aeb37c43a73ad3b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:56.417647 9377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea181a2a87634b5f9421f5cd13d32063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:56.417647 9236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea181a2a87634b5f9421f5cd13d32063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:56.420035 9517 tablet_service.cc:1467] Processing CreateTablet for tablet ea181a2a87634b5f9421f5cd13d32063 (DEFAULT_TABLE table=test-table [id=a6cd2e9f6c404ae2aeb37c43a73ad3b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:56.420317 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea181a2a87634b5f9421f5cd13d32063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:56.420331 9701 tablet_bootstrap.cc:492] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: Bootstrap starting.
I20260620 14:57:56.421180 9701 tablet_bootstrap.cc:654] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:56.422313 9701 tablet_bootstrap.cc:492] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: No bootstrap required, opened a new log
I20260620 14:57:56.422389 9701 ts_tablet_manager.cc:1397] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:57:56.422662 9704 tablet_bootstrap.cc:492] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: Bootstrap starting.
I20260620 14:57:56.423000 9702 tablet_bootstrap.cc:492] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: Bootstrap starting.
I20260620 14:57:56.423094 9701 raft_consensus.cc:348] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.423218 9701 raft_consensus.cc:374] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:56.423252 9701 raft_consensus.cc:729] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2916d65f7bc94872bdb053b937588e81, State: Initialized, Role: FOLLOWER
I20260620 14:57:56.423409 9701 consensus_queue.cc:260] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.423722 9704 tablet_bootstrap.cc:654] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:56.423794 9701 ts_tablet_manager.cc:1428] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:56.423920 9702 tablet_bootstrap.cc:654] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:56.425053 9704 tablet_bootstrap.cc:492] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: No bootstrap required, opened a new log
I20260620 14:57:56.425148 9704 ts_tablet_manager.cc:1397] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:56.425609 9704 raft_consensus.cc:348] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.425737 9704 raft_consensus.cc:374] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:56.425782 9704 raft_consensus.cc:729] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29d5723317b4f67be03e3573835d1cd, State: Initialized, Role: FOLLOWER
I20260620 14:57:56.425912 9704 consensus_queue.cc:260] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.426287 9704 ts_tablet_manager.cc:1428] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:56.426784 9702 tablet_bootstrap.cc:492] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: No bootstrap required, opened a new log
I20260620 14:57:56.426869 9702 ts_tablet_manager.cc:1397] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:57:56.427291 9702 raft_consensus.cc:348] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.427405 9702 raft_consensus.cc:374] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:56.427453 9702 raft_consensus.cc:729] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Initialized, Role: FOLLOWER
I20260620 14:57:56.427580 9702 consensus_queue.cc:260] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.427955 9702 ts_tablet_manager.cc:1428] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:56.457892 9671 raft_consensus.cc:482] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:56.457995 9671 raft_consensus.cc:504] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.458403 9671 leader_election.cc:288] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333), f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:56.458796 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea181a2a87634b5f9421f5cd13d32063" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2916d65f7bc94872bdb053b937588e81" is_pre_election: true
I20260620 14:57:56.458793 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea181a2a87634b5f9421f5cd13d32063" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd" is_pre_election: true
I20260620 14:57:56.458945 9537 raft_consensus.cc:2454] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 0.
I20260620 14:57:56.458947 9397 raft_consensus.cc:2454] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 0.
I20260620 14:57:56.459319 9189 leader_election.cc:302] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efbe10cd8969432bbf558b81bd76c46c, f29d5723317b4f67be03e3573835d1cd; no voters:
I20260620 14:57:56.459493 9671 raft_consensus.cc:2781] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:56.459549 9671 raft_consensus.cc:482] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:56.459591 9671 raft_consensus.cc:3037] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:56.460554 9671 raft_consensus.cc:504] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.460902 9671 leader_election.cc:288] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 election: Requested vote from peers 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333), f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:56.461217 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea181a2a87634b5f9421f5cd13d32063" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd"
I20260620 14:57:56.461217 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea181a2a87634b5f9421f5cd13d32063" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2916d65f7bc94872bdb053b937588e81"
I20260620 14:57:56.461330 9537 raft_consensus.cc:3037] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:56.461351 9397 raft_consensus.cc:3037] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:56.462374 9537 raft_consensus.cc:2454] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 1.
I20260620 14:57:56.462374 9397 raft_consensus.cc:2454] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 1.
I20260620 14:57:56.462639 9191 leader_election.cc:302] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2916d65f7bc94872bdb053b937588e81, efbe10cd8969432bbf558b81bd76c46c; no voters:
I20260620 14:57:56.462837 9671 raft_consensus.cc:2781] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:56.462951 9671 raft_consensus.cc:686] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 1 LEADER]: Becoming Leader. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Running, Role: LEADER
I20260620 14:57:56.463088 9671 consensus_queue.cc:237] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.464599 9099 catalog_manager.cc:5495] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c reported cstate change: term changed from 0 to 1, leader changed from <none> to efbe10cd8969432bbf558b81bd76c46c (127.7.46.129). New cstate: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } health_report { overall_health: UNKNOWN } } }
I20260620 14:57:56.489163 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:56.485893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.7.46.129:42985)
0620 14:57:56.486141 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:56.486153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:56.486165 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:57:56.486205 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:56.486212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:56.486251 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:56.486340 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:56.486613 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:56.487136 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:56.488095 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:56.488329 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:56.488439 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:56.488711 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:56.488786 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:56.488866 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:56.488956 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:57:56.489001 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260620 14:57:56.494635 9537 raft_consensus.cc:1261] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Refusing update from remote peer efbe10cd8969432bbf558b81bd76c46c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:56.494635 9397 raft_consensus.cc:1261] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Refusing update from remote peer efbe10cd8969432bbf558b81bd76c46c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:56.494985 9712 consensus_queue.cc:1035] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:56.495113 9676 consensus_queue.cc:1035] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:56.525784 9099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:56.525915 9099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:56.527715 9099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: Sending DeleteTablet for 3 replicas of tablet ea181a2a87634b5f9421f5cd13d32063
I20260620 14:57:56.528232 9377 tablet_service.cc:1514] Processing DeleteTablet for tablet ea181a2a87634b5f9421f5cd13d32063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:56 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:52326
I20260620 14:57:56.528236 9237 tablet_service.cc:1514] Processing DeleteTablet for tablet ea181a2a87634b5f9421f5cd13d32063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:56 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:35570
I20260620 14:57:56.528317 9517 tablet_service.cc:1514] Processing DeleteTablet for tablet ea181a2a87634b5f9421f5cd13d32063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:56 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:45314
I20260620 14:57:56.528403 9690 tablet_replica.cc:330] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: stopping tablet replica
I20260620 14:57:56.528403 9691 tablet_replica.cc:330] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: stopping tablet replica
I20260620 14:57:56.528505 9691 raft_consensus.cc:2229] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:56.528508 9690 raft_consensus.cc:2229] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:56.528626 9691 raft_consensus.cc:2258] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:56.528635 9692 tablet_replica.cc:330] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: stopping tablet replica
I20260620 14:57:56.528702 9690 raft_consensus.cc:2258] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:56.528739 9692 raft_consensus.cc:2229] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:56.528841 9692 raft_consensus.cc:2258] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:56.528913 9099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:57:56.529317 9099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:56.529325 9691 ts_tablet_manager.cc:1905] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:56.529356 9690 ts_tablet_manager.cc:1905] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:56.529803 9692 ts_tablet_manager.cc:1905] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:56.531373 9690 ts_tablet_manager.cc:1918] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:56.531433 9690 log.cc:1196] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/ea181a2a87634b5f9421f5cd13d32063
I20260620 14:57:56.531635 9690 ts_tablet_manager.cc:1939] T ea181a2a87634b5f9421f5cd13d32063 P efbe10cd8969432bbf558b81bd76c46c: Deleting consensus metadata
I20260620 14:57:56.531819 9691 ts_tablet_manager.cc:1918] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:56.531891 9691 log.cc:1196] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/ea181a2a87634b5f9421f5cd13d32063
I20260620 14:57:56.532141 9691 ts_tablet_manager.cc:1939] T ea181a2a87634b5f9421f5cd13d32063 P 2916d65f7bc94872bdb053b937588e81: Deleting consensus metadata
I20260620 14:57:56.532177 9692 ts_tablet_manager.cc:1918] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:57:56.532292 9692 log.cc:1196] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/ea181a2a87634b5f9421f5cd13d32063
I20260620 14:57:56.532267 9085 catalog_manager.cc:4841] TS efbe10cd8969432bbf558b81bd76c46c (127.7.46.129:42985): tablet ea181a2a87634b5f9421f5cd13d32063 (table test-table [id=a6cd2e9f6c404ae2aeb37c43a73ad3b8]) successfully deleted
I20260620 14:57:56.532531 9692 ts_tablet_manager.cc:1939] T ea181a2a87634b5f9421f5cd13d32063 P f29d5723317b4f67be03e3573835d1cd: Deleting consensus metadata
I20260620 14:57:56.533146 9085 catalog_manager.cc:4841] TS f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879): tablet ea181a2a87634b5f9421f5cd13d32063 (table test-table [id=a6cd2e9f6c404ae2aeb37c43a73ad3b8]) successfully deleted
I20260620 14:57:56.533334 9087 catalog_manager.cc:4841] TS 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333): tablet ea181a2a87634b5f9421f5cd13d32063 (table test-table [id=a6cd2e9f6c404ae2aeb37c43a73ad3b8]) successfully deleted
I20260620 14:57:56.535806 9237 tablet_service.cc:1467] Processing CreateTablet for tablet 3409e6aec27e4b34b9b8b843ed3fd492 (DEFAULT_TABLE table=test-table [id=049c6414e8ee4511b1c944bc2afa6e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:56.535857 9377 tablet_service.cc:1467] Processing CreateTablet for tablet 3409e6aec27e4b34b9b8b843ed3fd492 (DEFAULT_TABLE table=test-table [id=049c6414e8ee4511b1c944bc2afa6e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:56.535956 9517 tablet_service.cc:1467] Processing CreateTablet for tablet 3409e6aec27e4b34b9b8b843ed3fd492 (DEFAULT_TABLE table=test-table [id=049c6414e8ee4511b1c944bc2afa6e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:56.536048 9237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3409e6aec27e4b34b9b8b843ed3fd492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:56.536101 9377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3409e6aec27e4b34b9b8b843ed3fd492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:56.536160 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3409e6aec27e4b34b9b8b843ed3fd492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:56.537755 9702 tablet_bootstrap.cc:492] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c: Bootstrap starting.
I20260620 14:57:56.538108 9701 tablet_bootstrap.cc:492] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: Bootstrap starting.
I20260620 14:57:56.538143 9704 tablet_bootstrap.cc:492] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: Bootstrap starting.
I20260620 14:57:56.538519 9702 tablet_bootstrap.cc:654] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:56.538859 9704 tablet_bootstrap.cc:654] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:56.538892 9701 tablet_bootstrap.cc:654] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:56.539642 9702 tablet_bootstrap.cc:492] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c: No bootstrap required, opened a new log
I20260620 14:57:56.539721 9702 ts_tablet_manager.cc:1397] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:56.539898 9701 tablet_bootstrap.cc:492] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: No bootstrap required, opened a new log
I20260620 14:57:56.539976 9701 ts_tablet_manager.cc:1397] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:56.540126 9702 raft_consensus.cc:348] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.540236 9702 raft_consensus.cc:374] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:56.540282 9702 raft_consensus.cc:729] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Initialized, Role: FOLLOWER
I20260620 14:57:56.540354 9701 raft_consensus.cc:348] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.540405 9702 consensus_queue.cc:260] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.540457 9701 raft_consensus.cc:374] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:56.540514 9701 raft_consensus.cc:729] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2916d65f7bc94872bdb053b937588e81, State: Initialized, Role: FOLLOWER
I20260620 14:57:56.540632 9701 consensus_queue.cc:260] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.540683 9702 ts_tablet_manager.cc:1428] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:56.540879 9701 ts_tablet_manager.cc:1428] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:57:56.541824 9704 tablet_bootstrap.cc:492] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: No bootstrap required, opened a new log
I20260620 14:57:56.541918 9704 ts_tablet_manager.cc:1397] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:57:56.542378 9704 raft_consensus.cc:348] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.542501 9704 raft_consensus.cc:374] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:56.542536 9704 raft_consensus.cc:729] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29d5723317b4f67be03e3573835d1cd, State: Initialized, Role: FOLLOWER
I20260620 14:57:56.542642 9704 consensus_queue.cc:260] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.542905 9704 ts_tablet_manager.cc:1428] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:57:56.654054 9671 raft_consensus.cc:482] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:56.654193 9671 raft_consensus.cc:504] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.654678 9671 leader_election.cc:288] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333), f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:56.655074 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3409e6aec27e4b34b9b8b843ed3fd492" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2916d65f7bc94872bdb053b937588e81" is_pre_election: true
I20260620 14:57:56.655086 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3409e6aec27e4b34b9b8b843ed3fd492" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd" is_pre_election: true
I20260620 14:57:56.655231 9397 raft_consensus.cc:2454] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 0.
I20260620 14:57:56.655246 9537 raft_consensus.cc:2454] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 0.
I20260620 14:57:56.655570 9189 leader_election.cc:302] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efbe10cd8969432bbf558b81bd76c46c, f29d5723317b4f67be03e3573835d1cd; no voters:
I20260620 14:57:56.655751 9671 raft_consensus.cc:2781] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:56.655823 9671 raft_consensus.cc:482] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:56.655850 9671 raft_consensus.cc:3037] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:56.656816 9671 raft_consensus.cc:504] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.657104 9671 leader_election.cc:288] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 election: Requested vote from peers 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333), f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879)
I20260620 14:57:56.657406 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3409e6aec27e4b34b9b8b843ed3fd492" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d5723317b4f67be03e3573835d1cd"
I20260620 14:57:56.657406 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3409e6aec27e4b34b9b8b843ed3fd492" candidate_uuid: "efbe10cd8969432bbf558b81bd76c46c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2916d65f7bc94872bdb053b937588e81"
I20260620 14:57:56.657523 9397 raft_consensus.cc:3037] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:56.657523 9537 raft_consensus.cc:3037] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:56.658577 9397 raft_consensus.cc:2454] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 1.
I20260620 14:57:56.658576 9537 raft_consensus.cc:2454] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efbe10cd8969432bbf558b81bd76c46c in term 1.
I20260620 14:57:56.658859 9189 leader_election.cc:302] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2916d65f7bc94872bdb053b937588e81, efbe10cd8969432bbf558b81bd76c46c, f29d5723317b4f67be03e3573835d1cd; no voters:
I20260620 14:57:56.659047 9671 raft_consensus.cc:2781] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:56.659158 9671 raft_consensus.cc:686] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 1 LEADER]: Becoming Leader. State: Replica: efbe10cd8969432bbf558b81bd76c46c, State: Running, Role: LEADER
I20260620 14:57:56.659296 9671 consensus_queue.cc:237] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } }
I20260620 14:57:56.660768 9098 catalog_manager.cc:5495] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c reported cstate change: term changed from 0 to 1, leader changed from <none> to efbe10cd8969432bbf558b81bd76c46c (127.7.46.129). New cstate: current_term: 1 leader_uuid: "efbe10cd8969432bbf558b81bd76c46c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbe10cd8969432bbf558b81bd76c46c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 } health_report { overall_health: UNKNOWN } } }
I20260620 14:57:56.700870 9537 raft_consensus.cc:1261] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Refusing update from remote peer efbe10cd8969432bbf558b81bd76c46c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:56.701263 9397 raft_consensus.cc:1261] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Refusing update from remote peer efbe10cd8969432bbf558b81bd76c46c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:56.701323 9648 consensus_queue.cc:1035] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29d5723317b4f67be03e3573835d1cd" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 32879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:56.701640 9676 consensus_queue.cc:1035] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2916d65f7bc94872bdb053b937588e81" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:57:56.714290 9669 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:57:58.260833 9099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:58.260994 9099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:58.263412 9099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a13fd78d28dc47af840b2dd4217fbbaf: Sending DeleteTablet for 3 replicas of tablet 3409e6aec27e4b34b9b8b843ed3fd492
I20260620 14:57:58.264056 9377 tablet_service.cc:1514] Processing DeleteTablet for tablet 3409e6aec27e4b34b9b8b843ed3fd492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:58 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:52326
I20260620 14:57:58.264070 9237 tablet_service.cc:1514] Processing DeleteTablet for tablet 3409e6aec27e4b34b9b8b843ed3fd492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:58 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:35570
I20260620 14:57:58.264110 9517 tablet_service.cc:1514] Processing DeleteTablet for tablet 3409e6aec27e4b34b9b8b843ed3fd492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:58 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:45314
I20260620 14:57:58.264397 9740 tablet_replica.cc:330] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: stopping tablet replica
I20260620 14:57:58.264430 9741 tablet_replica.cc:330] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c: stopping tablet replica
I20260620 14:57:58.264518 9740 raft_consensus.cc:2229] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:58.264598 9741 raft_consensus.cc:2229] T 3409e6aec27e4b34b9b8b843ed3fd492 P efbe10cd8969432bbf558b81bd76c46c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:58.264631 9740 raft_consensus.cc:2258] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:58.264742 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 9172
I20260620 14:57:58.266065 9740 ts_tablet_manager.cc:1905] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:58.266263 9742 tablet_replica.cc:330] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: stopping tablet replica
I20260620 14:57:58.266417 9742 raft_consensus.cc:2229] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:58.266566 9742 raft_consensus.cc:2258] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:58.267189 9742 ts_tablet_manager.cc:1905] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:58.269495 9740 ts_tablet_manager.cc:1918] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:57:58.269572 9740 log.cc:1196] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/3409e6aec27e4b34b9b8b843ed3fd492
I20260620 14:57:58.269833 9740 ts_tablet_manager.cc:1939] T 3409e6aec27e4b34b9b8b843ed3fd492 P 2916d65f7bc94872bdb053b937588e81: Deleting consensus metadata
I20260620 14:57:58.269858 9742 ts_tablet_manager.cc:1918] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:57:58.269928 9742 log.cc:1196] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/3409e6aec27e4b34b9b8b843ed3fd492
I20260620 14:57:58.270213 9742 ts_tablet_manager.cc:1939] T 3409e6aec27e4b34b9b8b843ed3fd492 P f29d5723317b4f67be03e3573835d1cd: Deleting consensus metadata
I20260620 14:57:58.270526 9087 catalog_manager.cc:4841] TS 2916d65f7bc94872bdb053b937588e81 (127.7.46.130:38333): tablet 3409e6aec27e4b34b9b8b843ed3fd492 (table test-table [id=049c6414e8ee4511b1c944bc2afa6e2d]) successfully deleted
I20260620 14:57:58.272010 9085 catalog_manager.cc:4841] TS f29d5723317b4f67be03e3573835d1cd (127.7.46.131:32879): tablet 3409e6aec27e4b34b9b8b843ed3fd492 (table test-table [id=049c6414e8ee4511b1c944bc2afa6e2d]) successfully deleted
W20260620 14:57:58.273550 9085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:42985 (error 108)
I20260620 14:57:58.274688 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 9312
I20260620 14:57:58.274704 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:58.274252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42985 (local address 127.0.0.1:35590)
0620 14:57:58.274503 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:58.274614 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
W20260620 14:57:58.276031 9085 catalog_manager.cc:4584] TS efbe10cd8969432bbf558b81bd76c46c (127.7.46.129:42985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3409e6aec27e4b34b9b8b843ed3fd492: Network error: Client connection negotiation failed: client connection to 127.7.46.129:42985: connect: Connection refused (error 111)
I20260620 14:57:58.281332 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 9452
I20260620 14:57:58.287341 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 9068
2026-06-20T14:57:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6352 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:57:58 dist-test-slave-30t4 krb5kdc[9747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:57:58 dist-test-slave-30t4 krb5kdc[9747](info): set up 2 sockets
Jun 20 14:57:58 dist-test-slave-30t4 krb5kdc[9747](info): commencing operation
krb5kdc: starting...
W20260620 14:57:59.343643 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:57:59 dist-test-slave-30t4 krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:57:59.456668 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:46179
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:46179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:59.562362 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:59.562595 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:59.562633 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:59.566529 9760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:59.566582 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:59.566601 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:59.566618 9760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:59.566633 9760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:59.571862 9760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:46179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:46179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:59.572834 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:59.573814 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:59.582129 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:59.582166 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:59.582185 9760 server_base.cc:1034] running on GCE node
W20260620 14:57:59.582124 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:59.582921 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:59.584018 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:59.585177 9760 hybrid_clock.cc:648] HybridClock initialized: now 1781967479585164 us; error 26 us; skew 500 ppm
Jun 20 14:57:59 dist-test-slave-30t4 krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:59.587702 9760 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:57:59.588500 9760 webserver.cc:460] Webserver started at http://127.7.46.190:43699/ using document root <none> and password file <none>
I20260620 14:57:59.589020 9760 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:59.589105 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:59.589318 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:59.591244 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "71c530114c4e4b78afbe1437e2b6f880"
format_stamp: "Formatted at 2026-06-20 14:57:59 on dist-test-slave-30t4"
server_key: "b9a1f9c9ed980a79279ae52c4c98ba47"
server_key_iv: "75102489e6fe5637a0d1151166ed2f93"
server_key_version: "encryptionkey@0"
I20260620 14:57:59.591836 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "71c530114c4e4b78afbe1437e2b6f880"
format_stamp: "Formatted at 2026-06-20 14:57:59 on dist-test-slave-30t4"
server_key: "b9a1f9c9ed980a79279ae52c4c98ba47"
server_key_iv: "75102489e6fe5637a0d1151166ed2f93"
server_key_version: "encryptionkey@0"
I20260620 14:57:59.595258 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:57:59.597553 9775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:59.598708 9760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:57:59.598835 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "71c530114c4e4b78afbe1437e2b6f880"
format_stamp: "Formatted at 2026-06-20 14:57:59 on dist-test-slave-30t4"
server_key: "b9a1f9c9ed980a79279ae52c4c98ba47"
server_key_iv: "75102489e6fe5637a0d1151166ed2f93"
server_key_version: "encryptionkey@0"
I20260620 14:57:59.598935 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:59.612601 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:59.615156 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:59.615339 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:59.622063 9760 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:46179
I20260620 14:57:59.622088 9827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:46179 every 8 connection(s)
I20260620 14:57:59.622923 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:59.625519 9828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:59.630590 9828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880: Bootstrap starting.
I20260620 14:57:59.632121 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 9760
I20260620 14:57:59.632225 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:57:59.632509 7354 external_mini_cluster.cc:1428] Setting key 938bd3e3c7b220530db0cf0666b2906d
I20260620 14:57:59.633132 9828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:59.633994 9828 log.cc:824] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:59.636040 9828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880: No bootstrap required, opened a new log
I20260620 14:57:59.640121 9828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c530114c4e4b78afbe1437e2b6f880" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46179 } }
I20260620 14:57:59.640389 9828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:59.640455 9828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c530114c4e4b78afbe1437e2b6f880, State: Initialized, Role: FOLLOWER
I20260620 14:57:59.640844 9828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c530114c4e4b78afbe1437e2b6f880" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46179 } }
Jun 20 14:57:59 dist-test-slave-30t4 krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:57:59.640980 9828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:59.641044 9828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:59.641163 9828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:59.641963 9828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c530114c4e4b78afbe1437e2b6f880" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46179 } }
I20260620 14:57:59.642313 9828 leader_election.cc:302] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c530114c4e4b78afbe1437e2b6f880; no voters:
I20260620 14:57:59.642644 9828 leader_election.cc:288] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:57:59.642767 9833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:59.642998 9833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [term 1 LEADER]: Becoming Leader. State: Replica: 71c530114c4e4b78afbe1437e2b6f880, State: Running, Role: LEADER
I20260620 14:57:59.643323 9833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c530114c4e4b78afbe1437e2b6f880" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46179 } }
I20260620 14:57:59.643918 9828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:59.644423 9834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c530114c4e4b78afbe1437e2b6f880. Latest consensus state: current_term: 1 leader_uuid: "71c530114c4e4b78afbe1437e2b6f880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c530114c4e4b78afbe1437e2b6f880" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46179 } } }
I20260620 14:57:59.644548 9834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:59.645110 9835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c530114c4e4b78afbe1437e2b6f880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c530114c4e4b78afbe1437e2b6f880" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46179 } } }
I20260620 14:57:59.645206 9835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:59.646086 9838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:59.646790 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:59.633537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.7.46.190:46179)
0620 14:57:59.633915 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:59.633930 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:59.633962 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:57:59.634561 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:59.634576 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:59.634876 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:59.635131 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:59.635989 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:59.636956 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:59.638101 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:59.638544 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:59.638661 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:59.641602 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:59.641607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:59.641618 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:59.641642 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:59.643703 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:59.644215 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:59.644220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:59.644225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:59.644364 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:59.644672 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:59.644675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:59.644677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:59.645205 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:59.645409 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:59.645669 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:57:59.645964 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":121,"threads_started":1}
I20260620 14:57:59.649294 9838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:59.654059 9838 catalog_manager.cc:1347] Generated new cluster ID: c464d6f73c704ec7973a317381c56369
I20260620 14:57:59.654116 9838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:57:59.679862 9838 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:57:59.681107 9838 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:57:59.691502 9838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c530114c4e4b78afbe1437e2b6f880: Generated new TSK 0
I20260620 14:57:59.692312 9838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:57:59.720309 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:46179
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:59.829963 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:59.830204 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:59.830245 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:59.834081 9854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:59.834131 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:59.834187 9854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:57:59.839085 9854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:57:59.839989 9854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:59.840884 9854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:59.850281 9859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:59.850287 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:59.850287 9860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:59.850525 9854 server_base.cc:1034] running on GCE node
I20260620 14:57:59.850883 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:59.851473 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:59.852635 9854 hybrid_clock.cc:648] HybridClock initialized: now 1781967479852591 us; error 52 us; skew 500 ppm
Jun 20 14:57:59 dist-test-slave-30t4 krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:59.855271 9854 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:57:59.856173 9854 webserver.cc:460] Webserver started at http://127.7.46.129:34569/ using document root <none> and password file <none>
I20260620 14:57:59.856706 9854 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:59.856765 9854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:59.856964 9854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:59.858870 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "a5cf54402a5040fc96cd89a3a8c311d0"
format_stamp: "Formatted at 2026-06-20 14:57:59 on dist-test-slave-30t4"
server_key: "58aaea5f5c65c040495304337a696f09"
server_key_iv: "b2af45b7c47f8dc5effbfe3616f7008d"
server_key_version: "encryptionkey@0"
I20260620 14:57:59.859408 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "a5cf54402a5040fc96cd89a3a8c311d0"
format_stamp: "Formatted at 2026-06-20 14:57:59 on dist-test-slave-30t4"
server_key: "58aaea5f5c65c040495304337a696f09"
server_key_iv: "b2af45b7c47f8dc5effbfe3616f7008d"
server_key_version: "encryptionkey@0"
I20260620 14:57:59.862890 9854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:57:59.865185 9869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:59.866319 9854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:57:59.866495 9854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "a5cf54402a5040fc96cd89a3a8c311d0"
format_stamp: "Formatted at 2026-06-20 14:57:59 on dist-test-slave-30t4"
server_key: "58aaea5f5c65c040495304337a696f09"
server_key_iv: "b2af45b7c47f8dc5effbfe3616f7008d"
server_key_version: "encryptionkey@0"
I20260620 14:57:59.866611 9854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:59.888598 9854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:59.891484 9854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:59.891685 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:59.892271 9854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:59.893070 9854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:59.893139 9854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:59.893208 9854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:59.893244 9854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:59.901768 9854 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:37443
I20260620 14:57:59.901801 9982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:37443 every 8 connection(s)
I20260620 14:57:59.902640 9854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:59.906186 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 9854
I20260620 14:57:59.906301 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:57:59.906566 7354 external_mini_cluster.cc:1428] Setting key 7280c075764fea6a63792e1950434523
Jun 20 14:57:59 dist-test-slave-30t4 krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 14:57:59.917177 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:59.904409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:54161 (local address 127.7.46.190:46179)
0620 14:57:59.904548 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:59.904554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:59.905392 (+ 838us) server_negotiation.cc:407] Connection header received
0620 14:57:59.906999 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:59.907003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:59.907062 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:59.907151 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:59.908551 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:59.909127 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:59.910392 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:59.910832 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:59.910956 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:59.913549 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:59.913556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:59.913560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:59.913595 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:59.915241 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:59.915722 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:59.915729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:59.915734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:59.915847 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:59.916180 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:59.916183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:59.916185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:59.916425 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:59.916525 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:59.916965 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 14:57:59.917061 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:57:59.917958 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:59.904833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:46179 (local address 127.7.46.129:54161)
0620 14:57:59.905249 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:59.905289 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:59.906771 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:59.907337 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:59.907344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:59.907744 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:59.908386 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:59.908409 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:59.909257 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:59.909264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:59.910272 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:59.910281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:59.910986 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:59.910988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:59.911122 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:59.911697 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:59.911712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:59.913382 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:59.915356 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:59.915361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:59.915372 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:59.915617 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:59.915954 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:59.915956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:59.915958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:59.916088 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:59.916539 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:59.916543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:59.916757 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 14:57:59.916908 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:57:59.917146 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 14:57:59.919143 9983 heartbeater.cc:347] Connected to a master server at 127.7.46.190:46179
I20260620 14:57:59.919375 9983 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:59.919895 9983 heartbeater.cc:511] Master 127.7.46.190:46179 requested a full tablet report, sending...
I20260620 14:57:59.921556 9792 ts_manager.cc:194] Registered new tserver with Master: a5cf54402a5040fc96cd89a3a8c311d0 (127.7.46.129:37443)
I20260620 14:57:59.923172 9792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:54161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:57:59.934363 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:46179
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:00.041172 9988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:00.041421 9988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:00.041463 9988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:00.045529 9988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:00.045581 9988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:00.045701 9988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:00.050705 9988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:00.051594 9988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:00.052578 9988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:00.062228 9996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:00.062213 9993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:00.062527 9988 server_base.cc:1034] running on GCE node
W20260620 14:58:00.062215 9994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:00.062938 9988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:00.063556 9988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:00.064754 9988 hybrid_clock.cc:648] HybridClock initialized: now 1781967480064714 us; error 56 us; skew 500 ppm
Jun 20 14:58:00 dist-test-slave-30t4 krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:00.067646 9988 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:58:00.068535 9988 webserver.cc:460] Webserver started at http://127.7.46.130:42949/ using document root <none> and password file <none>
I20260620 14:58:00.069175 9988 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:00.069279 9988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:00.069543 9988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:00.071760 9988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "b20a5873280c41ca8e89073f7834e8ef"
format_stamp: "Formatted at 2026-06-20 14:58:00 on dist-test-slave-30t4"
server_key: "d3e887eeee735639f7dd5fff08cdc0b3"
server_key_iv: "2dbe8b3163f7839d82f2052f11186b50"
server_key_version: "encryptionkey@0"
I20260620 14:58:00.072424 9988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "b20a5873280c41ca8e89073f7834e8ef"
format_stamp: "Formatted at 2026-06-20 14:58:00 on dist-test-slave-30t4"
server_key: "d3e887eeee735639f7dd5fff08cdc0b3"
server_key_iv: "2dbe8b3163f7839d82f2052f11186b50"
server_key_version: "encryptionkey@0"
I20260620 14:58:00.076161 9988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:00.078555 10003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:00.079901 9988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:00.080029 9988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "b20a5873280c41ca8e89073f7834e8ef"
format_stamp: "Formatted at 2026-06-20 14:58:00 on dist-test-slave-30t4"
server_key: "d3e887eeee735639f7dd5fff08cdc0b3"
server_key_iv: "2dbe8b3163f7839d82f2052f11186b50"
server_key_version: "encryptionkey@0"
I20260620 14:58:00.080132 9988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:00.105898 9988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:00.109126 9988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:00.109314 9988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:00.109915 9988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:00.110693 9988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:00.110767 9988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:00.110836 9988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:00.110883 9988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:00.119342 9988 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:38335
I20260620 14:58:00.119370 10116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:38335 every 8 connection(s)
I20260620 14:58:00.120239 9988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:58:00 dist-test-slave-30t4 krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:00.130146 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 9988
I20260620 14:58:00.130273 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:00.130633 7354 external_mini_cluster.cc:1428] Setting key f9c2adc4c4597c13ddf775d522e7ea99
I20260620 14:58:00.133232 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:00.121901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:42415 (local address 127.7.46.190:46179)
0620 14:58:00.122037 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:00.122041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:00.122834 (+ 793us) server_negotiation.cc:407] Connection header received
0620 14:58:00.124182 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:00.124185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:00.124242 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:00.124321 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:00.125407 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:00.125925 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:00.127020 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:00.127254 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:00.127363 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:00.129642 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:00.129650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:00.129652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:00.129674 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:00.131276 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:00.131787 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:00.131793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:00.131795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:00.131873 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:00.132203 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:00.132208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:00.132209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:00.132378 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:00.132483 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:00.132940 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 14:58:00.133071 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:00.134023 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:00.122258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:46179 (local address 127.7.46.130:42415)
0620 14:58:00.122687 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:00.122728 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:00.123981 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:00.124477 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:00.124485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:00.124815 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:00.125264 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:00.125276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:00.126043 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:00.126047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:00.126895 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:00.126904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:00.127407 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:00.127411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:00.127574 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:00.128123 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:00.128138 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:00.129470 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:00.131387 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:00.131392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:00.131404 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:00.131654 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:00.131970 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:00.131973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:00.131975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:00.132109 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:00.132481 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:00.132488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:00.132720 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 14:58:00.132900 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 14:58:00.133187 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260620 14:58:00.135089 10117 heartbeater.cc:347] Connected to a master server at 127.7.46.190:46179
I20260620 14:58:00.135324 10117 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:00.135818 10117 heartbeater.cc:511] Master 127.7.46.190:46179 requested a full tablet report, sending...
I20260620 14:58:00.136842 9792 ts_manager.cc:194] Registered new tserver with Master: b20a5873280c41ca8e89073f7834e8ef (127.7.46.130:38335)
I20260620 14:58:00.137701 9792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:42415
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:58:00.156587 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:46179
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:00.264488 10122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:00.264750 10122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:00.264823 10122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:00.268838 10122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:00.268908 10122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:00.268991 10122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:00.274170 10122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:00.275171 10122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:00.276196 10122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:00.285591 10127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:00.285591 10128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:00.285645 10122 server_base.cc:1034] running on GCE node
W20260620 14:58:00.285610 10130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:00.286182 10122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:00.286754 10122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:00.287932 10122 hybrid_clock.cc:648] HybridClock initialized: now 1781967480287916 us; error 26 us; skew 500 ppm
Jun 20 14:58:00 dist-test-slave-30t4 krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:00.290494 10122 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:58:00.291329 10122 webserver.cc:460] Webserver started at http://127.7.46.131:35927/ using document root <none> and password file <none>
I20260620 14:58:00.291926 10122 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:00.292011 10122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:00.292243 10122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:00.294159 10122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "a40ff74bbace4ef5ac36198e5f042116"
format_stamp: "Formatted at 2026-06-20 14:58:00 on dist-test-slave-30t4"
server_key: "151a579e0e1008e11ca35e141536acd5"
server_key_iv: "74470adea81ce8c2471a134b8c130d89"
server_key_version: "encryptionkey@0"
I20260620 14:58:00.294708 10122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "a40ff74bbace4ef5ac36198e5f042116"
format_stamp: "Formatted at 2026-06-20 14:58:00 on dist-test-slave-30t4"
server_key: "151a579e0e1008e11ca35e141536acd5"
server_key_iv: "74470adea81ce8c2471a134b8c130d89"
server_key_version: "encryptionkey@0"
I20260620 14:58:00.298251 10122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:58:00.300581 10137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:00.301751 10122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:00.301868 10122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "a40ff74bbace4ef5ac36198e5f042116"
format_stamp: "Formatted at 2026-06-20 14:58:00 on dist-test-slave-30t4"
server_key: "151a579e0e1008e11ca35e141536acd5"
server_key_iv: "74470adea81ce8c2471a134b8c130d89"
server_key_version: "encryptionkey@0"
I20260620 14:58:00.301972 10122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:00.322326 10122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:00.325430 10122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:00.325621 10122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:00.326197 10122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:00.326985 10122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:00.327064 10122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:00.327132 10122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:00.327184 10122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:00.336360 10122 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:33233
I20260620 14:58:00.336387 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:33233 every 8 connection(s)
I20260620 14:58:00.337208 10122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:00.342042 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 10122
I20260620 14:58:00.342166 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:00.342460 7354 external_mini_cluster.cc:1428] Setting key 3f307db4243a22cb3689743e3f1c86ff
Jun 20 14:58:00 dist-test-slave-30t4 krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:00.350474 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:00.338935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:47241 (local address 127.7.46.190:46179)
0620 14:58:00.339066 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:00.339070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:00.339916 (+ 846us) server_negotiation.cc:407] Connection header received
0620 14:58:00.341270 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:00.341273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:00.341331 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:00.341448 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:00.342906 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:00.343493 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:00.344904 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:00.345121 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:00.345203 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:00.347361 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:00.347364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:00.347366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:00.347389 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:00.348843 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:00.349251 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:00.349253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:00.349255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:00.349320 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:00.349591 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:00.349593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:00.349594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:00.349750 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:00.349820 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:00.350267 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 14:58:00.350363 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:00.351209 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:00.339300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:46179 (local address 127.7.46.131:47241)
0620 14:58:00.339720 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:00.339759 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:00.341057 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:00.341606 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:00.341613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:00.342099 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:00.342753 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:00.342782 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:00.343622 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:00.343626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:00.344807 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:00.344816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:00.345245 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:00.345247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:00.345383 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:00.345878 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:00.345893 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:00.347223 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:00.348933 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:00.348937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:00.348948 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:00.349171 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:00.349394 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:00.349396 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:00.349397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:00.349511 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:00.349828 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:00.349834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:00.350030 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 14:58:00.350180 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 14:58:00.350397 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20260620 14:58:00.352324 10251 heartbeater.cc:347] Connected to a master server at 127.7.46.190:46179
I20260620 14:58:00.352528 10251 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:00.352926 10251 heartbeater.cc:511] Master 127.7.46.190:46179 requested a full tablet report, sending...
I20260620 14:58:00.353816 9792 ts_manager.cc:194] Registered new tserver with Master: a40ff74bbace4ef5ac36198e5f042116 (127.7.46.131:33233)
I20260620 14:58:00.354317 9792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:47241
I20260620 14:58:00.356173 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:58:00.368060 9831 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:00.366707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.7.46.190:46179)
0620 14:58:00.366822 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:00.366826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:00.367018 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:58:00.367122 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:00.367124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:00.367161 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:00.367227 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:00.367937 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59270: BlockingRecv error: recv got EOF from 127.0.0.1:59270 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:58:00.369499 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 9854
I20260620 14:58:00.374962 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 9988
I20260620 14:58:00.379933 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 10122
I20260620 14:58:00.384708 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 9760
2026-06-20T14:58:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2096 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:00 dist-test-slave-30t4 krb5kdc[10263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:00 dist-test-slave-30t4 krb5kdc[10263](info): set up 2 sockets
Jun 20 14:58:00 dist-test-slave-30t4 krb5kdc[10263](info): commencing operation
krb5kdc: starting...
W20260620 14:58:01.427663 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:01 dist-test-slave-30t4 krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:58:01.546940 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34429
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:01.653030 10276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:01.653264 10276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:01.653302 10276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:01.657320 10276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:01.657378 10276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:01.657398 10276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:01.657414 10276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:01.657429 10276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:01.662441 10276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:34429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:01.663403 10276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:01.664433 10276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:01.672523 10281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:01.672536 10282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:01.672611 10284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:01.672832 10276 server_base.cc:1034] running on GCE node
I20260620 14:58:01.673362 10276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:01.674427 10276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:01.675590 10276 hybrid_clock.cc:648] HybridClock initialized: now 1781967481675560 us; error 41 us; skew 500 ppm
Jun 20 14:58:01 dist-test-slave-30t4 krb5kdc[10263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:01.683247 10276 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:58:01.684232 10276 webserver.cc:460] Webserver started at http://127.7.46.190:44065/ using document root <none> and password file <none>
I20260620 14:58:01.684772 10276 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:01.684831 10276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:01.685074 10276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:01.687032 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "25993ea7c8704459884db4eeec93219b"
format_stamp: "Formatted at 2026-06-20 14:58:01 on dist-test-slave-30t4"
server_key: "e8e713703f4f9d534b8ee17599ede6c5"
server_key_iv: "a27fcc33ed94d80214e75e4944ccfa2c"
server_key_version: "encryptionkey@0"
I20260620 14:58:01.687588 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "25993ea7c8704459884db4eeec93219b"
format_stamp: "Formatted at 2026-06-20 14:58:01 on dist-test-slave-30t4"
server_key: "e8e713703f4f9d534b8ee17599ede6c5"
server_key_iv: "a27fcc33ed94d80214e75e4944ccfa2c"
server_key_version: "encryptionkey@0"
I20260620 14:58:01.691220 10276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:58:01.693538 10291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:01.694741 10276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:01.694837 10276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "25993ea7c8704459884db4eeec93219b"
format_stamp: "Formatted at 2026-06-20 14:58:01 on dist-test-slave-30t4"
server_key: "e8e713703f4f9d534b8ee17599ede6c5"
server_key_iv: "a27fcc33ed94d80214e75e4944ccfa2c"
server_key_version: "encryptionkey@0"
I20260620 14:58:01.694965 10276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:01.709194 10276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:01.711872 10276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:01.712061 10276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:01.718631 10276 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:34429
I20260620 14:58:01.718909 10343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:34429 every 8 connection(s)
I20260620 14:58:01.719558 10276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:01.722195 10344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:01.722433 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 10276
I20260620 14:58:01.722541 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:01.722836 7354 external_mini_cluster.cc:1428] Setting key c2cd395a1565b77961a4cb5fb3c7ccef
I20260620 14:58:01.728801 10344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b: Bootstrap starting.
I20260620 14:58:01.731462 10344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b: Neither blocks nor log segments found. Creating new log.
Jun 20 14:58:01 dist-test-slave-30t4 krb5kdc[10263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:01.732231 10344 log.cc:824] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:01.734506 10344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b: No bootstrap required, opened a new log
I20260620 14:58:01.737605 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:01.724116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.7.46.190:34429)
0620 14:58:01.724507 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:01.724521 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:01.724555 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:58:01.725301 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:01.725316 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:01.725633 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:01.725894 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:01.726783 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:01.727718 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:01.728925 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:01.729382 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:01.729492 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:01.732366 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:01.732375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:01.732393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:01.732428 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:01.734475 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:01.735017 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:01.735023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:01.735030 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:01.735165 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:01.735420 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:01.735423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:01.735424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:01.735919 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:01.736122 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:01.736312 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:58:01.736595 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
I20260620 14:58:01.739125 10344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25993ea7c8704459884db4eeec93219b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34429 } }
I20260620 14:58:01.739396 10344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:01.739506 10344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25993ea7c8704459884db4eeec93219b, State: Initialized, Role: FOLLOWER
I20260620 14:58:01.739989 10344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25993ea7c8704459884db4eeec93219b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34429 } }
I20260620 14:58:01.740141 10344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:01.740226 10344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:01.740322 10344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:01.740427 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:01.741179 10344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25993ea7c8704459884db4eeec93219b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34429 } }
I20260620 14:58:01.741504 10344 leader_election.cc:302] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25993ea7c8704459884db4eeec93219b; no voters:
I20260620 14:58:01.741856 10344 leader_election.cc:288] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:01.741907 10349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:01.742090 10349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [term 1 LEADER]: Becoming Leader. State: Replica: 25993ea7c8704459884db4eeec93219b, State: Running, Role: LEADER
I20260620 14:58:01.742430 10349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25993ea7c8704459884db4eeec93219b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34429 } }
I20260620 14:58:01.743064 10344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:01.743532 10349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25993ea7c8704459884db4eeec93219b. Latest consensus state: current_term: 1 leader_uuid: "25993ea7c8704459884db4eeec93219b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25993ea7c8704459884db4eeec93219b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34429 } } }
I20260620 14:58:01.743712 10349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:01.744030 10349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25993ea7c8704459884db4eeec93219b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25993ea7c8704459884db4eeec93219b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34429 } } }
I20260620 14:58:01.744174 10349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b [sys.catalog]: This master's current role is: LEADER
W20260620 14:58:01.748145 10366 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:58:01.748222 10366 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:58:01.748278 10357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:01.749081 10357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:01.754194 10357 catalog_manager.cc:1347] Generated new cluster ID: c3f2ca04a4844a2190ab0515d324f956
I20260620 14:58:01.754249 10357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:01.762025 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:01.756609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.7.46.190:34429)
0620 14:58:01.756715 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:01.756719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:01.756964 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:58:01.757058 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:01.757061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:01.757110 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:01.757201 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:01.757721 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:01.758537 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:01.759647 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:01.760058 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:01.760164 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:01.761051 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:01.761055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:01.761058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:01.761581 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:01.761617 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:01.761636 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:01.761725 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:01.761825 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:58:01.761882 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:58:01.774315 10357 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:01.775527 10357 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:01.782428 10357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25993ea7c8704459884db4eeec93219b: Generated new TSK 0
I20260620 14:58:01.783186 10357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:01.787401 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 10276
2026-06-20T14:58:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1400 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:01 dist-test-slave-30t4 krb5kdc[10377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:01 dist-test-slave-30t4 krb5kdc[10377](info): set up 2 sockets
Jun 20 14:58:01 dist-test-slave-30t4 krb5kdc[10377](info): commencing operation
krb5kdc: starting...
W20260620 14:58:02.823824 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:02 dist-test-slave-30t4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:58:02.943964 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37325
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:37325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:03.050287 10390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:03.050567 10390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:03.050634 10390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:03.054641 10390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:03.054721 10390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:03.054759 10390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:03.054816 10390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:03.054857 10390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:03.059968 10390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:37325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:03.060953 10390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:03.061932 10390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:03.070096 10396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:03.070104 10395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:03.070252 10390 server_base.cc:1034] running on GCE node
W20260620 14:58:03.070384 10398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:03.070873 10390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:03.072009 10390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:03.073191 10390 hybrid_clock.cc:648] HybridClock initialized: now 1781967483073176 us; error 27 us; skew 500 ppm
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:03.075738 10390 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:58:03.076611 10390 webserver.cc:460] Webserver started at http://127.7.46.190:45551/ using document root <none> and password file <none>
I20260620 14:58:03.077162 10390 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:03.077255 10390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:03.077478 10390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:03.079454 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "500bbe5b3f9b44dfac014afae6f1b51a"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "1c851edc279f86ad2f83eb7362c7d611"
server_key_iv: "e94da1c9b9468e592621dfc3354e50ab"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.080057 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "500bbe5b3f9b44dfac014afae6f1b51a"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "1c851edc279f86ad2f83eb7362c7d611"
server_key_iv: "e94da1c9b9468e592621dfc3354e50ab"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.083575 10390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:58:03.085961 10405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.087215 10390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:03.087352 10390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "500bbe5b3f9b44dfac014afae6f1b51a"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "1c851edc279f86ad2f83eb7362c7d611"
server_key_iv: "e94da1c9b9468e592621dfc3354e50ab"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.087462 10390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:03.110558 10390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:03.113503 10390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:03.113718 10390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:03.120348 10390 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:37325
I20260620 14:58:03.120631 10457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:37325 every 8 connection(s)
I20260620 14:58:03.121342 10390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:03.124070 10458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:03.129118 10458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a: Bootstrap starting.
I20260620 14:58:03.129832 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 10390
I20260620 14:58:03.129951 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:03.130219 7354 external_mini_cluster.cc:1428] Setting key 36af34f60db5ac8705a9c15948edfc3b
I20260620 14:58:03.131963 10458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:03.132663 10458 log.cc:824] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:03.134724 10458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a: No bootstrap required, opened a new log
I20260620 14:58:03.138530 10458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bbe5b3f9b44dfac014afae6f1b51a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37325 } }
I20260620 14:58:03.138760 10458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:03.138844 10458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500bbe5b3f9b44dfac014afae6f1b51a, State: Initialized, Role: FOLLOWER
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:03.139317 10458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bbe5b3f9b44dfac014afae6f1b51a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37325 } }
I20260620 14:58:03.139472 10458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:03.139561 10458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:03.139659 10458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:03.140848 10458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bbe5b3f9b44dfac014afae6f1b51a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37325 } }
I20260620 14:58:03.141196 10458 leader_election.cc:302] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500bbe5b3f9b44dfac014afae6f1b51a; no voters:
I20260620 14:58:03.141541 10458 leader_election.cc:288] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:03.141649 10463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:03.141865 10463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [term 1 LEADER]: Becoming Leader. State: Replica: 500bbe5b3f9b44dfac014afae6f1b51a, State: Running, Role: LEADER
I20260620 14:58:03.142200 10463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bbe5b3f9b44dfac014afae6f1b51a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37325 } }
I20260620 14:58:03.142843 10458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:03.143782 10465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500bbe5b3f9b44dfac014afae6f1b51a. Latest consensus state: current_term: 1 leader_uuid: "500bbe5b3f9b44dfac014afae6f1b51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bbe5b3f9b44dfac014afae6f1b51a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37325 } } }
I20260620 14:58:03.143903 10465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:03.144219 10464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500bbe5b3f9b44dfac014afae6f1b51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bbe5b3f9b44dfac014afae6f1b51a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37325 } } }
I20260620 14:58:03.144312 10464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:03.144659 10472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:03.145757 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:03.131419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.7.46.190:37325)
0620 14:58:03.131847 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:03.131861 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:03.131902 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:58:03.132546 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:03.132562 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:03.132927 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:03.133201 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:03.133938 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.135385 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.136264 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.136976 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.137131 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:03.139619 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:03.139625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:03.139640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:03.139673 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:03.141673 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.142193 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.142198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.142203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.142325 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.142683 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.142687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.142688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.143184 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:03.144100 (+ 916us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:03.144258 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:58:03.144578 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":126,"threads_started":1}
I20260620 14:58:03.147594 10472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:03.152654 10472 catalog_manager.cc:1347] Generated new cluster ID: 29077cc2c07542c3a8f58be3296e17ba
I20260620 14:58:03.152729 10472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:03.162798 10472 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:03.163646 10472 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:03.170981 10472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500bbe5b3f9b44dfac014afae6f1b51a: Generated new TSK 0
I20260620 14:58:03.171598 10472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:58:03.200008 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37325
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:03.309392 10484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:03.309626 10484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:03.309666 10484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:03.313601 10484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:03.313659 10484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:03.313745 10484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:03.318984 10484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:03.320081 10484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:03.321079 10484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:03.330476 10489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:03.330476 10492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:03.330607 10484 server_base.cc:1034] running on GCE node
W20260620 14:58:03.330472 10490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:03.331121 10484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:03.331760 10484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:03.332998 10484 hybrid_clock.cc:648] HybridClock initialized: now 1781967483332975 us; error 33 us; skew 500 ppm
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:03.335718 10484 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:58:03.336562 10484 webserver.cc:460] Webserver started at http://127.7.46.129:41079/ using document root <none> and password file <none>
I20260620 14:58:03.337109 10484 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:03.337198 10484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:03.337417 10484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:03.339360 10484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "c39f66d12fd94b409fbd96a3b084c30c"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "ece5f8ec575740e6ab7b2881ebd2f7b1"
server_key_iv: "9d250c18ae526d12e7beed2b72341aee"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.339972 10484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "c39f66d12fd94b409fbd96a3b084c30c"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "ece5f8ec575740e6ab7b2881ebd2f7b1"
server_key_iv: "9d250c18ae526d12e7beed2b72341aee"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.343670 10484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:58:03.346060 10499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.347208 10484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:03.347337 10484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "c39f66d12fd94b409fbd96a3b084c30c"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "ece5f8ec575740e6ab7b2881ebd2f7b1"
server_key_iv: "9d250c18ae526d12e7beed2b72341aee"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.347445 10484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:03.371439 10484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:03.374269 10484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:03.374475 10484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:03.375046 10484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:03.375847 10484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:03.375916 10484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.375984 10484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:03.376031 10484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.384523 10484 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:39333
I20260620 14:58:03.384549 10612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:39333 every 8 connection(s)
I20260620 14:58:03.385345 10484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:03.385815 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 10484
I20260620 14:58:03.385962 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:03.386286 7354 external_mini_cluster.cc:1428] Setting key c6cfd2c67d7d6acc815102abc1f8dd9b
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:03.404450 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:03.387082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:43273 (local address 127.7.46.190:37325)
0620 14:58:03.387221 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:03.387225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:03.388251 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 14:58:03.390035 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:03.390039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:03.390096 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:03.390324 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:03.391984 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.392733 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.394603 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.394847 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.394937 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:03.400203 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:03.400211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:03.400217 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:03.400253 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:03.401813 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.402456 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.402458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.402459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.402537 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.403110 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.403113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.403114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.403298 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:03.403453 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:03.404219 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 14:58:03.404327 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:03.405740 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:03.387537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37325 (local address 127.7.46.129:43273)
0620 14:58:03.388034 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:03.388108 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:03.389823 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:03.390522 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:03.390535 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:03.391093 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:03.391804 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:03.391829 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.392877 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.392881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:03.394449 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:03.394462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.395029 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.395034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:03.395253 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:03.396013 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:03.396039 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:03.400014 (+ 3975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:03.401938 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:03.401947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:03.401968 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:03.402338 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:03.402625 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:03.402631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:03.402636 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:03.402831 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:03.403468 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:03.403477 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:03.403881 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 14:58:03.404104 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 14:58:03.404526 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
I20260620 14:58:03.407148 10613 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37325
I20260620 14:58:03.407405 10613 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:03.407946 10613 heartbeater.cc:511] Master 127.7.46.190:37325 requested a full tablet report, sending...
I20260620 14:58:03.409530 10422 ts_manager.cc:194] Registered new tserver with Master: c39f66d12fd94b409fbd96a3b084c30c (127.7.46.129:39333)
I20260620 14:58:03.410678 10422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:43273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:58:03.414726 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37325
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:03.524874 10618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:03.525116 10618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:03.525161 10618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:03.529923 10618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:03.529976 10618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:03.530035 10618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:03.535161 10618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:03.536206 10618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:03.537248 10618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:03.547224 10626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:03.547274 10624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:03.547180 10623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:03.547538 10618 server_base.cc:1034] running on GCE node
I20260620 14:58:03.548072 10618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:03.548671 10618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:03.549825 10618 hybrid_clock.cc:648] HybridClock initialized: now 1781967483549780 us; error 56 us; skew 500 ppm
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:03.552568 10618 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:58:03.553330 10618 webserver.cc:460] Webserver started at http://127.7.46.130:32943/ using document root <none> and password file <none>
I20260620 14:58:03.553823 10618 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:03.553880 10618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:03.554041 10618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:03.556042 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "86c8564b5b1e4fc787f1de9494797b84"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "5510ec7403eb95bc85340894a7fdbb3b"
server_key_iv: "ffd4dcd27577499c71dc26c97b4277a3"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.556524 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "86c8564b5b1e4fc787f1de9494797b84"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "5510ec7403eb95bc85340894a7fdbb3b"
server_key_iv: "ffd4dcd27577499c71dc26c97b4277a3"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.559981 10618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:03.562265 10633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.563542 10618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:03.563647 10618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "86c8564b5b1e4fc787f1de9494797b84"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "5510ec7403eb95bc85340894a7fdbb3b"
server_key_iv: "ffd4dcd27577499c71dc26c97b4277a3"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.563735 10618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:03.574833 10618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:03.578191 10618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:03.578359 10618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:03.578904 10618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:03.579690 10618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:03.579748 10618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.579862 10618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:03.579893 10618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.588994 10618 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:39243
I20260620 14:58:03.589089 10746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:39243 every 8 connection(s)
I20260620 14:58:03.589915 10618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:03.591061 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 10618
I20260620 14:58:03.591171 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:03.591497 7354 external_mini_cluster.cc:1428] Setting key 7f3ac65e29c1bf96af1e22be8dd79111
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
I20260620 14:58:03.604996 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:03.591912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:55069 (local address 127.7.46.190:37325)
0620 14:58:03.592053 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:03.592059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:03.593215 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 14:58:03.594557 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:03.594561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:03.594623 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:03.594708 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:03.595897 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.596431 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.598302 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.598539 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.598627 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:03.601365 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:03.601368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:03.601389 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:03.601419 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:03.602840 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.603374 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.603377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.603379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.603465 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.603863 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.603870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.603875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.604104 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:03.604217 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:03.604709 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 14:58:03.604821 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:58:03.605840 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:03.592482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37325 (local address 127.7.46.130:55069)
0620 14:58:03.593071 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:03.593128 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:03.594340 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:03.594894 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:03.594900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:03.595242 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:03.595729 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:03.595751 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.596564 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.596570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:03.598146 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:03.598161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.598704 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.598710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:03.598942 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:03.599548 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:03.599567 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:03.601197 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:03.602945 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:03.602949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:03.602959 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:03.603272 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:03.603572 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:03.603575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:03.603577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:03.603721 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:03.604217 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:03.604225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:03.604466 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 14:58:03.604652 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:58:03.604968 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":172,"threads_started":1}
I20260620 14:58:03.607190 10747 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37325
I20260620 14:58:03.607457 10747 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:03.608023 10747 heartbeater.cc:511] Master 127.7.46.190:37325 requested a full tablet report, sending...
I20260620 14:58:03.609095 10422 ts_manager.cc:194] Registered new tserver with Master: 86c8564b5b1e4fc787f1de9494797b84 (127.7.46.130:39243)
I20260620 14:58:03.609655 10422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:55069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:58:03.618533 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37325
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:03.724217 10752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:03.724475 10752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:03.724517 10752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:03.728358 10752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:03.728402 10752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:03.728458 10752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:03.733465 10752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:03.734390 10752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:03.735437 10752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:03.744630 10758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:03.744643 10760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:03.744637 10757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:03.745023 10752 server_base.cc:1034] running on GCE node
I20260620 14:58:03.745342 10752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:03.745919 10752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:03.747107 10752 hybrid_clock.cc:648] HybridClock initialized: now 1781967483747080 us; error 47 us; skew 500 ppm
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:03.749771 10752 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:58:03.750658 10752 webserver.cc:460] Webserver started at http://127.7.46.131:36381/ using document root <none> and password file <none>
I20260620 14:58:03.751199 10752 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:03.751257 10752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:03.751487 10752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:03.753650 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "211e23d7c30d42dfb4129da17eb8d457"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "464368fca6979670cb6b0b5d40f23b71"
server_key_iv: "73eb499d3640cd6d8c187fc2431bec2a"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.754204 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "211e23d7c30d42dfb4129da17eb8d457"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "464368fca6979670cb6b0b5d40f23b71"
server_key_iv: "73eb499d3640cd6d8c187fc2431bec2a"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.757875 10752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:03.760317 10767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.761631 10752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:03.761757 10752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "211e23d7c30d42dfb4129da17eb8d457"
format_stamp: "Formatted at 2026-06-20 14:58:03 on dist-test-slave-30t4"
server_key: "464368fca6979670cb6b0b5d40f23b71"
server_key_iv: "73eb499d3640cd6d8c187fc2431bec2a"
server_key_version: "encryptionkey@0"
I20260620 14:58:03.761864 10752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:03.780460 10752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:03.783231 10752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:03.783430 10752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:03.784094 10752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:03.784883 10752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:03.784927 10752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.784991 10752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:03.785037 10752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:03.793668 10752 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:34515
I20260620 14:58:03.793689 10880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:34515 every 8 connection(s)
I20260620 14:58:03.794598 10752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:03.804478 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 10752
I20260620 14:58:03.804625 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:03.804908 7354 external_mini_cluster.cc:1428] Setting key 6c6942d68cbdbc5ae14121776ad8115b
I20260620 14:58:03.807999 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:03.796300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:47923 (local address 127.7.46.190:37325)
0620 14:58:03.796464 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:03.796469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:03.797246 (+ 777us) server_negotiation.cc:407] Connection header received
0620 14:58:03.798517 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:03.798520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:03.798567 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:03.798645 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:03.799721 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.800320 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.801378 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.801602 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.801683 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:03.804087 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:03.804090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:03.804092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:03.804117 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:03.805856 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.806339 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.806341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.806342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.806419 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.806776 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.806778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.806779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.807043 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:03.807153 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:03.807659 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 14:58:03.807819 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:58:03.808753 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:03.796674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37325 (local address 127.7.46.131:47923)
0620 14:58:03.797099 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:03.797139 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:03.798302 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:03.798797 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:03.798803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:03.799128 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:03.799569 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:03.799581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.800428 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.800430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:03.801267 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:03.801275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.801725 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.801728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:03.801859 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:03.802370 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:03.802385 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:03.803925 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:03.805962 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:03.805968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:03.805983 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:03.806236 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:03.806511 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:03.806514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:03.806515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:03.806684 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:03.807150 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:03.807156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:03.807413 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 14:58:03.807616 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:58:03.807957 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260620 14:58:03.809792 10881 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37325
I20260620 14:58:03.810036 10881 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:03.810524 10881 heartbeater.cc:511] Master 127.7.46.190:37325 requested a full tablet report, sending...
I20260620 14:58:03.811519 10422 ts_manager.cc:194] Registered new tserver with Master: 211e23d7c30d42dfb4129da17eb8d457 (127.7.46.131:34515)
I20260620 14:58:03.812140 10422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:47923
I20260620 14:58:03.818769 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967483, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:03.843032 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:03.834771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.7.46.190:37325)
0620 14:58:03.834936 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:03.834942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:03.835095 (+ 153us) server_negotiation.cc:407] Connection header received
0620 14:58:03.835274 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:03.835276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:03.835317 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:03.835404 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:03.836149 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.836669 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.837506 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:03.837714 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:03.837820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:03.840119 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:03.840127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:03.840129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:03.840151 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:03.841356 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.841753 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.841758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.841762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.841840 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:03.842110 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:03.842115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:03.842117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:03.842387 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:03.842525 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:03.842754 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:58:03.842868 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 14:58:03.843394 10422 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36166
I20260620 14:58:03.844688 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 10484
I20260620 14:58:03.849946 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 10618
I20260620 14:58:03.855522 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 10752
I20260620 14:58:03.860562 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 10390
2026-06-20T14:58:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2074 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10893](info): set up 2 sockets
Jun 20 14:58:03 dist-test-slave-30t4 krb5kdc[10893](info): commencing operation
krb5kdc: starting...
W20260620 14:58:04.891731 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:04 dist-test-slave-30t4 krb5kdc[10893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:58:05.004801 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:42957
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:05.114066 10906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:05.114315 10906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:05.114355 10906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:05.118386 10906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:05.118443 10906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:05.118462 10906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:05.118479 10906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:05.118520 10906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:05.124459 10906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:05.125501 10906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:05.126500 10906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:05.134857 10914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:05.135001 10906 server_base.cc:1034] running on GCE node
W20260620 14:58:05.134829 10912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:05.134846 10911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:05.135756 10906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:05.136816 10906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:05.138010 10906 hybrid_clock.cc:648] HybridClock initialized: now 1781967485137980 us; error 46 us; skew 500 ppm
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:05.140703 10906 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:58:05.141534 10906 webserver.cc:460] Webserver started at http://127.7.46.190:39057/ using document root <none> and password file <none>
I20260620 14:58:05.142076 10906 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:05.142133 10906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:05.142287 10906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:05.144276 10906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "93439555abde49c99d80937784026d55"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "1c542e18f39c652008592471d0cfc645"
server_key_iv: "c14b246e1ee5ff86fcac6d06c1c19efd"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.144773 10906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "93439555abde49c99d80937784026d55"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "1c542e18f39c652008592471d0cfc645"
server_key_iv: "c14b246e1ee5ff86fcac6d06c1c19efd"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.148448 10906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:05.150775 10921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.152053 10906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:05.152189 10906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "93439555abde49c99d80937784026d55"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "1c542e18f39c652008592471d0cfc645"
server_key_iv: "c14b246e1ee5ff86fcac6d06c1c19efd"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.152289 10906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:05.163208 10906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:05.166167 10906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:05.166364 10906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:05.173336 10906 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:42957
I20260620 14:58:05.173362 10973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:42957 every 8 connection(s)
I20260620 14:58:05.174254 10906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:05.176905 10974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:05.180709 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 10906
I20260620 14:58:05.180840 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:05.181119 7354 external_mini_cluster.cc:1428] Setting key 367e0432d9b64f0a22730e5bfae5ec6f
I20260620 14:58:05.182760 10974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55: Bootstrap starting.
I20260620 14:58:05.184931 10974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:05.185658 10974 log.cc:824] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:05.187438 10974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55: No bootstrap required, opened a new log
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:05.191139 10974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93439555abde49c99d80937784026d55" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42957 } }
I20260620 14:58:05.191386 10974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:05.191454 10974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93439555abde49c99d80937784026d55, State: Initialized, Role: FOLLOWER
I20260620 14:58:05.191964 10974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93439555abde49c99d80937784026d55" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42957 } }
I20260620 14:58:05.192126 10974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:05.192211 10974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:05.192327 10974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:05.193133 10974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93439555abde49c99d80937784026d55" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42957 } }
I20260620 14:58:05.193457 10974 leader_election.cc:302] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93439555abde49c99d80937784026d55; no voters:
I20260620 14:58:05.193763 10974 leader_election.cc:288] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:05.193850 10979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:05.194103 10979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [term 1 LEADER]: Becoming Leader. State: Replica: 93439555abde49c99d80937784026d55, State: Running, Role: LEADER
I20260620 14:58:05.194438 10979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93439555abde49c99d80937784026d55" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42957 } }
I20260620 14:58:05.195070 10974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:05.195614 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.182511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.7.46.190:42957)
0620 14:58:05.182988 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:05.183003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:05.183038 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:58:05.183665 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:05.183681 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:05.184084 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:05.184414 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:05.185252 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.186468 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.187311 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.187899 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.188076 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.190525 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:05.190531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:05.190544 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:05.190572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:05.192392 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.192817 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.192823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.192829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.192942 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.193222 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.193226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.193228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.193673 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:05.193875 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:05.194251 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 14:58:05.194706 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":132,"threads_started":1}
I20260620 14:58:05.195624 10979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93439555abde49c99d80937784026d55. Latest consensus state: current_term: 1 leader_uuid: "93439555abde49c99d80937784026d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93439555abde49c99d80937784026d55" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42957 } } }
I20260620 14:58:05.196049 10979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:05.196288 10979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93439555abde49c99d80937784026d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93439555abde49c99d80937784026d55" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42957 } } }
I20260620 14:58:05.196393 10979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:05.196906 10988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:05.200745 10988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:05.206219 10988 catalog_manager.cc:1347] Generated new cluster ID: 3d9bc4576f294f6ba5be6d840a7b90b9
I20260620 14:58:05.206281 10988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:05.223057 10988 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:05.225163 10988 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:05.233577 10988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93439555abde49c99d80937784026d55: Generated new TSK 0
I20260620 14:58:05.234357 10988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:58:05.266865 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42957
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:05.375303 11000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:05.375583 11000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:05.375658 11000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:05.379560 11000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:05.379643 11000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:05.379736 11000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:05.384739 11000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:05.385705 11000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:05.386673 11000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:05.395700 11008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:05.395701 11005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:05.395727 11006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:05.395977 11000 server_base.cc:1034] running on GCE node
I20260620 14:58:05.396427 11000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:05.397099 11000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:05.398276 11000 hybrid_clock.cc:648] HybridClock initialized: now 1781967485398251 us; error 42 us; skew 500 ppm
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:05.400835 11000 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:58:05.401680 11000 webserver.cc:460] Webserver started at http://127.7.46.129:46341/ using document root <none> and password file <none>
I20260620 14:58:05.402220 11000 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:05.402308 11000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:05.402530 11000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:05.404543 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "0f0e619dc9504474982a3ef47096f0a6"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "bd20a47c5d7d7033a240e19c46521e24"
server_key_iv: "d7b96ca22b70fc9bc81414309e83f97b"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.405088 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "0f0e619dc9504474982a3ef47096f0a6"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "bd20a47c5d7d7033a240e19c46521e24"
server_key_iv: "d7b96ca22b70fc9bc81414309e83f97b"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.408499 11000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:05.411222 11015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.412361 11000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:58:05.412488 11000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "0f0e619dc9504474982a3ef47096f0a6"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "bd20a47c5d7d7033a240e19c46521e24"
server_key_iv: "d7b96ca22b70fc9bc81414309e83f97b"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.412593 11000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:05.446645 11000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:05.449479 11000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:05.449688 11000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:05.450265 11000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:05.451074 11000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:05.451148 11000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.451217 11000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:05.451260 11000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.459971 11000 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:45153
I20260620 14:58:05.460047 11128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:45153 every 8 connection(s)
I20260620 14:58:05.460872 11000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:05.463105 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 11000
I20260620 14:58:05.463234 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:05.463533 7354 external_mini_cluster.cc:1428] Setting key 970a8e5677575a19886acbb66c78340e
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 14:58:05.475428 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.462628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:35597 (local address 127.7.46.190:42957)
0620 14:58:05.462764 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:05.462768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:05.463670 (+ 902us) server_negotiation.cc:407] Connection header received
0620 14:58:05.465440 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:05.465443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:05.465493 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:05.465571 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:05.466914 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.467430 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.468634 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.468859 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.469017 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.471748 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:05.471752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:05.471754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:05.471817 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:05.473301 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.473797 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.473802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.473804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.473883 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.474259 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.474262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.474264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.474443 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:05.474556 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:05.475090 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 14:58:05.475209 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 14:58:05.476399 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.463005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:42957 (local address 127.7.46.129:35597)
0620 14:58:05.463507 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:05.463558 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:05.465243 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:05.465742 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:05.465748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:05.466080 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:05.466760 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:05.466780 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.467547 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.467551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:05.468496 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:05.468505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.469065 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.469068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:05.469294 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.469923 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:05.469949 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:05.471572 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:05.473411 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:05.473417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:05.473432 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:05.473697 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:05.474001 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:05.474005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:05.474007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:05.474156 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:05.474545 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:05.474551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:05.474799 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 14:58:05.475042 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 14:58:05.475352 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":134,"threads_started":1}
I20260620 14:58:05.477720 11129 heartbeater.cc:347] Connected to a master server at 127.7.46.190:42957
I20260620 14:58:05.477962 11129 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:05.478435 11129 heartbeater.cc:511] Master 127.7.46.190:42957 requested a full tablet report, sending...
I20260620 14:58:05.479960 10938 ts_manager.cc:194] Registered new tserver with Master: 0f0e619dc9504474982a3ef47096f0a6 (127.7.46.129:45153)
I20260620 14:58:05.481053 10938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:35597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:58:05.489843 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42957
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:05.594981 11134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:05.595233 11134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:05.595309 11134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:05.599413 11134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:05.599493 11134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:05.599577 11134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:05.604604 11134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:05.605551 11134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:05.606483 11134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:05.615628 11142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:05.615640 11139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:05.615687 11140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:05.615837 11134 server_base.cc:1034] running on GCE node
I20260620 14:58:05.616312 11134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:05.616925 11134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:05.618114 11134 hybrid_clock.cc:648] HybridClock initialized: now 1781967485618099 us; error 25 us; skew 500 ppm
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:05.620872 11134 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:58:05.621670 11134 webserver.cc:460] Webserver started at http://127.7.46.130:41373/ using document root <none> and password file <none>
I20260620 14:58:05.622210 11134 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:05.622299 11134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:05.622517 11134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:05.624504 11134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "2b57a1ddf10341e8a44ee3de2d4525c3"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "7b96422753bbdbf01ae7ca83485c9241"
server_key_iv: "c5d6f38e2aeea9f324735086c85e1244"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.625058 11134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "2b57a1ddf10341e8a44ee3de2d4525c3"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "7b96422753bbdbf01ae7ca83485c9241"
server_key_iv: "c5d6f38e2aeea9f324735086c85e1244"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.628542 11134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:05.630771 11149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.631917 11134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:05.632040 11134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "2b57a1ddf10341e8a44ee3de2d4525c3"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "7b96422753bbdbf01ae7ca83485c9241"
server_key_iv: "c5d6f38e2aeea9f324735086c85e1244"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.632154 11134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:05.658564 11134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:05.661754 11134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:05.661957 11134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:05.662532 11134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:05.663332 11134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:05.663403 11134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.663470 11134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:05.663522 11134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.672425 11134 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:40705
I20260620 14:58:05.672550 11262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:40705 every 8 connection(s)
I20260620 14:58:05.673290 11134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:05.675519 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 11134
I20260620 14:58:05.675649 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:05.675982 7354 external_mini_cluster.cc:1428] Setting key 51bc680d7991f1da30cde0a96276b86b
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
I20260620 14:58:05.687965 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.674999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:60613 (local address 127.7.46.190:42957)
0620 14:58:05.675136 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:05.675140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:05.676075 (+ 935us) server_negotiation.cc:407] Connection header received
0620 14:58:05.677729 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:05.677732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:05.677773 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:05.677838 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:05.679388 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.680271 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.681355 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.681576 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.681699 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.684447 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:05.684451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:05.684453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:05.684474 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:05.685885 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.686384 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.686389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.686391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.686520 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.686881 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.686886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.686887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.687067 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:05.687175 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:05.687655 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 14:58:05.687792 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
I20260620 14:58:05.688797 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.675385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:42957 (local address 127.7.46.130:60613)
0620 14:58:05.675913 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:05.675968 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:05.677522 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:05.678031 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:05.678042 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:05.678544 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:05.679224 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:05.679248 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.680393 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.680395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:05.681250 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:05.681258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.681733 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.681740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:05.681971 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.682578 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:05.682603 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:05.684280 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:05.686006 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:05.686012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:05.686026 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:05.686279 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:05.686636 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:05.686641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:05.686643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:05.686779 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:05.687171 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:05.687177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:05.687412 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 14:58:05.687600 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:58:05.687925 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260620 14:58:05.689945 11263 heartbeater.cc:347] Connected to a master server at 127.7.46.190:42957
I20260620 14:58:05.690214 11263 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:05.690793 11263 heartbeater.cc:511] Master 127.7.46.190:42957 requested a full tablet report, sending...
I20260620 14:58:05.691915 10938 ts_manager.cc:194] Registered new tserver with Master: 2b57a1ddf10341e8a44ee3de2d4525c3 (127.7.46.130:40705)
I20260620 14:58:05.692767 10938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:60613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:58:05.703685 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42957
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:05.810098 11268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:05.810331 11268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:05.810374 11268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:05.814697 11268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:05.814749 11268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:05.814805 11268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:05.820061 11268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:05.820957 11268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:05.821883 11268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:05.830971 11274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:05.831036 11268 server_base.cc:1034] running on GCE node
W20260620 14:58:05.830997 11273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:05.831000 11276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:05.831580 11268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:05.832259 11268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:05.833424 11268 hybrid_clock.cc:648] HybridClock initialized: now 1781967485833406 us; error 32 us; skew 500 ppm
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:05.836381 11268 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:58:05.837152 11268 webserver.cc:460] Webserver started at http://127.7.46.131:45447/ using document root <none> and password file <none>
I20260620 14:58:05.837663 11268 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:05.837721 11268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:05.837877 11268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:05.839937 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "0fe22b7db1184286926d9fe76b83835d"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "d810064ade298ec91c05d9fc1f4b31a1"
server_key_iv: "2b8fca4312fa5712065054ff92d31e5d"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.840510 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "0fe22b7db1184286926d9fe76b83835d"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "d810064ade298ec91c05d9fc1f4b31a1"
server_key_iv: "2b8fca4312fa5712065054ff92d31e5d"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.844045 11268 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:05.846237 11283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.847417 11268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:05.847563 11268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "0fe22b7db1184286926d9fe76b83835d"
format_stamp: "Formatted at 2026-06-20 14:58:05 on dist-test-slave-30t4"
server_key: "d810064ade298ec91c05d9fc1f4b31a1"
server_key_iv: "2b8fca4312fa5712065054ff92d31e5d"
server_key_version: "encryptionkey@0"
I20260620 14:58:05.847671 11268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:05.875250 11268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:05.878716 11268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:05.878949 11268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:05.879570 11268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:05.880463 11268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:05.880533 11268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.880632 11268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:05.880684 11268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:05.889377 11268 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:32889
I20260620 14:58:05.889420 11396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:32889 every 8 connection(s)
I20260620 14:58:05.890252 11268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:05.899513 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 11268
I20260620 14:58:05.899660 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:05.899953 7354 external_mini_cluster.cc:1428] Setting key f23a2c60f403a4e3362ff3d635611b8b
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:05.904119 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.892025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:51299 (local address 127.7.46.190:42957)
0620 14:58:05.892178 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:05.892182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:05.892987 (+ 805us) server_negotiation.cc:407] Connection header received
0620 14:58:05.894344 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:05.894347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:05.894403 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:05.894510 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:05.895704 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.896339 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.897858 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.898076 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.898184 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.900830 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:05.900834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:05.900837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:05.900868 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:05.902339 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.902780 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.902782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.902783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.902851 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.903125 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.903127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.903128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.903282 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:05.903355 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:05.903891 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 14:58:05.903999 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:58:05.904870 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.892428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:42957 (local address 127.7.46.131:51299)
0620 14:58:05.892850 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:05.892890 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:05.894145 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:05.894688 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:05.894695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:05.895077 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:05.895571 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:05.895586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.896456 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.896462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:05.897738 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:05.897747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.898216 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.898219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:05.898385 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.898884 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:05.898900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:05.900683 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:05.902434 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:05.902438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:05.902449 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:05.902697 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:05.902926 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:05.902928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:05.902929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:05.903051 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:05.903364 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:05.903368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:05.903593 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 14:58:05.903742 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 14:58:05.904056 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260620 14:58:05.906052 11397 heartbeater.cc:347] Connected to a master server at 127.7.46.190:42957
I20260620 14:58:05.906268 11397 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:05.906687 11397 heartbeater.cc:511] Master 127.7.46.190:42957 requested a full tablet report, sending...
I20260620 14:58:05.907625 10938 ts_manager.cc:194] Registered new tserver with Master: 0fe22b7db1184286926d9fe76b83835d (127.7.46.131:32889)
I20260620 14:58:05.908259 10938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:51299
I20260620 14:58:05.913636 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:58:05.939088 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.929670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.7.46.129:45153)
0620 14:58:05.929958 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:05.929973 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:05.930001 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:58:05.930066 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:05.930075 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:05.930287 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:05.930442 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:05.931109 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.931936 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.932779 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.933146 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.933260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.935865 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:05.935874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:05.935894 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:05.935917 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:05.937462 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.937852 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.937856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.937858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.937931 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.938149 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.938151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.938152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.938371 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:05.938521 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:05.938700 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:58:05.938870 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
Jun 20 14:58:05 dist-test-slave-30t4 krb5kdc[10893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:05.949522 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:05.941803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.7.46.190:42957)
0620 14:58:05.941914 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:05.941917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:05.941934 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:58:05.942032 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:05.942034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:05.942072 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:05.942136 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:05.942696 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.943195 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.943974 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:05.944199 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:05.944288 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:05.946540 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:05.946544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:05.946551 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:05.946585 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:05.948221 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.948580 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.948582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.948584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.948656 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:05.948870 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:05.948872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:05.948873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:05.949063 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:05.949160 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:05.949312 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 14:58:05.949407 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 14:58:05.949941 10938 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32970
I20260620 14:58:05.950549 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 11000
I20260620 14:58:05.956838 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 11134
I20260620 14:58:05.961810 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 11268
I20260620 14:58:05.966835 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 10906
2026-06-20T14:58:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2106 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:06 dist-test-slave-30t4 krb5kdc[11409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:06 dist-test-slave-30t4 krb5kdc[11409](info): set up 2 sockets
Jun 20 14:58:06 dist-test-slave-30t4 krb5kdc[11409](info): commencing operation
krb5kdc: starting...
W20260620 14:58:07.007947 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:58:07.123487 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34125
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:34125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:07.229429 11422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:07.229703 11422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:07.229744 11422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:07.235105 11422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:07.235160 11422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:07.235180 11422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:07.235196 11422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:07.235247 11422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:07.240437 11422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:34125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:07.241420 11422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:07.242386 11422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:07.250254 11427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:07.250250 11430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:07.250391 11422 server_base.cc:1034] running on GCE node
W20260620 14:58:07.250259 11428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:07.251057 11422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:07.252164 11422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:07.253324 11422 hybrid_clock.cc:648] HybridClock initialized: now 1781967487253296 us; error 39 us; skew 500 ppm
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:07.260254 11422 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:58:07.260911 11422 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:07.260985 11422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:07.261148 11422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:07.263055 11422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "28b4b7f086ba4803b0f209d8e0a08011"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "d237cebb103c93cbfed838919c423fd8"
server_key_iv: "ae3b3ffa1548e837710931f4ba364b49"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.263545 11422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "28b4b7f086ba4803b0f209d8e0a08011"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "d237cebb103c93cbfed838919c423fd8"
server_key_iv: "ae3b3ffa1548e837710931f4ba364b49"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.267019 11422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:07.269289 11436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.270447 11422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:07.270545 11422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "28b4b7f086ba4803b0f209d8e0a08011"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "d237cebb103c93cbfed838919c423fd8"
server_key_iv: "ae3b3ffa1548e837710931f4ba364b49"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.270625 11422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:07.302412 11422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:07.305761 11422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:07.305932 11422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:07.313311 11422 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:34125
I20260620 14:58:07.313664 11488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:34125 every 8 connection(s)
I20260620 14:58:07.314025 11422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:07.316848 11489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:07.319200 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 11422
I20260620 14:58:07.319339 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:07.319661 7354 external_mini_cluster.cc:1428] Setting key f81de4913a16b9e1d4f212bbb66815f2
I20260620 14:58:07.324010 11489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011: Bootstrap starting.
I20260620 14:58:07.326188 11489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:07.327047 11489 log.cc:824] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:07.329006 11489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011: No bootstrap required, opened a new log
I20260620 14:58:07.332521 11489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b4b7f086ba4803b0f209d8e0a08011" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34125 } }
I20260620 14:58:07.332868 11489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:07.332949 11489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b4b7f086ba4803b0f209d8e0a08011, State: Initialized, Role: FOLLOWER
I20260620 14:58:07.333433 11489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b4b7f086ba4803b0f209d8e0a08011" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34125 } }
I20260620 14:58:07.333560 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:07.321127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.7.46.190:34125)
0620 14:58:07.321581 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:07.321596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:07.321632 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:58:07.322275 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:07.322291 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:07.322620 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:07.322955 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:07.323726 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.324770 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.325622 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.326089 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.326260 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:07.328836 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:07.328842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:07.328854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:07.328882 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:07.330627 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:07.331078 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:07.331089 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:07.331093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:07.331201 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:07.331499 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:07.331502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:07.331504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:07.331982 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:07.332205 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:07.332448 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:58:07.332737 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":146,"threads_started":1}
I20260620 14:58:07.333591 11489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:07.333735 11489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:07.333842 11489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:07.334725 11489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b4b7f086ba4803b0f209d8e0a08011" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34125 } }
I20260620 14:58:07.335062 11489 leader_election.cc:302] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b4b7f086ba4803b0f209d8e0a08011; no voters:
I20260620 14:58:07.335359 11489 leader_election.cc:288] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:07.335485 11494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:07.335703 11494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [term 1 LEADER]: Becoming Leader. State: Replica: 28b4b7f086ba4803b0f209d8e0a08011, State: Running, Role: LEADER
I20260620 14:58:07.336067 11494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b4b7f086ba4803b0f209d8e0a08011" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34125 } }
I20260620 14:58:07.336577 11489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:07.337152 11496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b4b7f086ba4803b0f209d8e0a08011. Latest consensus state: current_term: 1 leader_uuid: "28b4b7f086ba4803b0f209d8e0a08011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b4b7f086ba4803b0f209d8e0a08011" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34125 } } }
I20260620 14:58:07.337312 11496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:07.337600 11495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28b4b7f086ba4803b0f209d8e0a08011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b4b7f086ba4803b0f209d8e0a08011" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34125 } } }
I20260620 14:58:07.337749 11495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:07.337785 11502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:07.340804 11502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:07.349337 11502 catalog_manager.cc:1347] Generated new cluster ID: 78d0dc3d2b1e4f108c8580345686bc39
I20260620 14:58:07.349417 11502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:07.354043 11502 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:07.355208 11502 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:07.361368 11502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011: Generated new TSK 0
I20260620 14:58:07.362063 11502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:58:07.397218 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34125
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:07.505297 11515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:07.505570 11515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:07.505645 11515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:07.509663 11515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:07.509747 11515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:07.509830 11515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:07.515040 11515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:07.516093 11515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:07.517068 11515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:07.526430 11520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:07.526571 11521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:07.526440 11523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:07.526710 11515 server_base.cc:1034] running on GCE node
I20260620 14:58:07.527222 11515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:07.527922 11515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:07.529102 11515 hybrid_clock.cc:648] HybridClock initialized: now 1781967487529085 us; error 40 us; skew 500 ppm
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:07.531739 11515 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:58:07.532552 11515 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:07.532613 11515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:07.532773 11515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:07.534670 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "bdfd5182908747bb98a90c3d702dd4b9"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "3924f013d7962513303d865ba44bdeb5"
server_key_iv: "a1a83ad6dbbd08aa2d74791f74419e26"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.535233 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "bdfd5182908747bb98a90c3d702dd4b9"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "3924f013d7962513303d865ba44bdeb5"
server_key_iv: "a1a83ad6dbbd08aa2d74791f74419e26"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.538996 11515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:58:07.541261 11529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.542452 11515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:07.542593 11515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "bdfd5182908747bb98a90c3d702dd4b9"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "3924f013d7962513303d865ba44bdeb5"
server_key_iv: "a1a83ad6dbbd08aa2d74791f74419e26"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.542677 11515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:07.558686 11515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:07.561417 11515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:07.561561 11515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:07.562037 11515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:07.562800 11515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:07.562875 11515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.562917 11515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:07.562933 11515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.571666 11515 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:37613
I20260620 14:58:07.571683 11642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:37613 every 8 connection(s)
I20260620 14:58:07.572239 11515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:07.573127 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 11515
I20260620 14:58:07.573273 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:07.573611 7354 external_mini_cluster.cc:1428] Setting key 130eda39fdbc0f391a17ac718e61f49f
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 14:58:07.586030 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:07.574053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:48959 (local address 127.7.46.190:34125)
0620 14:58:07.574210 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:07.574215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:07.575146 (+ 931us) server_negotiation.cc:407] Connection header received
0620 14:58:07.576501 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:07.576504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:07.576558 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:07.576631 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:07.577743 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.578272 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.579482 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.579713 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.579867 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:07.582271 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:07.582274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:07.582276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:07.582299 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:07.583847 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:07.584422 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:07.584426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:07.584428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:07.584511 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:07.584890 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:07.584894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:07.584896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:07.585143 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:07.585246 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:07.585740 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 14:58:07.585857 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:58:07.586880 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:07.574514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34125 (local address 127.7.46.129:48959)
0620 14:58:07.574986 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:07.575036 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:07.576278 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:07.576800 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:07.576807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:07.577154 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:07.577600 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:07.577612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.578395 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.578401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:07.579349 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:07.579359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.579911 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.579915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:07.580076 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:07.580689 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:07.580704 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:07.582098 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:07.583959 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:07.583965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:07.583980 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:07.584261 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:07.584606 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:07.584610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:07.584612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:07.584746 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:07.585254 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:07.585262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:07.585494 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 14:58:07.585687 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 14:58:07.585994 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":136,"threads_started":1}
I20260620 14:58:07.588268 11643 heartbeater.cc:347] Connected to a master server at 127.7.46.190:34125
I20260620 14:58:07.588529 11643 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:07.589087 11643 heartbeater.cc:511] Master 127.7.46.190:34125 requested a full tablet report, sending...
I20260620 14:58:07.590801 11453 ts_manager.cc:194] Registered new tserver with Master: bdfd5182908747bb98a90c3d702dd4b9 (127.7.46.129:37613)
I20260620 14:58:07.592453 11453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:48959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:58:07.600384 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34125
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:07.707736 11648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:07.708035 11648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:07.708079 11648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:07.712042 11648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:07.712093 11648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:07.712152 11648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:07.717222 11648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:07.718142 11648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:07.719056 11648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:58:07.728222 11648 server_base.cc:1034] running on GCE node
W20260620 14:58:07.728192 11654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:07.728188 11656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:07.728186 11653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:07.728863 11648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:07.729471 11648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:07.730667 11648 hybrid_clock.cc:648] HybridClock initialized: now 1781967487730631 us; error 45 us; skew 500 ppm
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:07.733312 11648 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:58:07.734038 11648 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:07.734127 11648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:07.734349 11648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:07.736398 11648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "bf0be9d7614240ecb63b46241c237db9"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "402fdd86cb825d915a8ec3e9c5197ceb"
server_key_iv: "16c0687da17d896e83b88d8094f2392f"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.737011 11648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "bf0be9d7614240ecb63b46241c237db9"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "402fdd86cb825d915a8ec3e9c5197ceb"
server_key_iv: "16c0687da17d896e83b88d8094f2392f"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.740576 11648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:07.742767 11662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.743994 11648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 14:58:07.744117 11648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "bf0be9d7614240ecb63b46241c237db9"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "402fdd86cb825d915a8ec3e9c5197ceb"
server_key_iv: "16c0687da17d896e83b88d8094f2392f"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.744221 11648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:07.762567 11648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:07.765295 11648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:07.765494 11648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:07.766033 11648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:07.766815 11648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:07.766894 11648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.766989 11648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:07.767042 11648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.776233 11648 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:40977
I20260620 14:58:07.776299 11775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:40977 every 8 connection(s)
I20260620 14:58:07.776813 11648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:07.786203 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 11648
I20260620 14:58:07.786346 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:07.786729 7354 external_mini_cluster.cc:1428] Setting key 6a05f7ace1a877bb70a4e9c3ef3356c1
I20260620 14:58:07.789764 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:07.778523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:59403 (local address 127.7.46.190:34125)
0620 14:58:07.778652 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:07.778656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:07.779447 (+ 791us) server_negotiation.cc:407] Connection header received
0620 14:58:07.780708 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:07.780711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:07.780756 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:07.780853 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:07.781876 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.782401 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.783474 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.783683 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.783833 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:07.786212 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:07.786219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:07.786225 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:07.786257 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:07.787768 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:07.788325 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:07.788329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:07.788332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:07.788415 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:07.788742 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:07.788747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:07.788749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:07.788929 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:07.789034 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:07.789499 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 14:58:07.789617 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:58:07.790623 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:07.778891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34125 (local address 127.7.46.130:59403)
0620 14:58:07.779295 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:07.779335 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:07.780504 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:07.780957 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:07.780963 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:07.781285 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:07.781716 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:07.781728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.782524 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.782527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:07.783358 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:07.783366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.783893 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.783900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:07.784140 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:07.784620 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:07.784635 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:07.786035 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:07.787938 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:07.787944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:07.787957 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:07.788216 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:07.788506 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:07.788511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:07.788513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:07.788647 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:07.789032 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:07.789039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:07.789267 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 14:58:07.789456 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 14:58:07.789885 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":102,"threads_started":1}
I20260620 14:58:07.791706 11776 heartbeater.cc:347] Connected to a master server at 127.7.46.190:34125
I20260620 14:58:07.791947 11776 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:07.792373 11776 heartbeater.cc:511] Master 127.7.46.190:34125 requested a full tablet report, sending...
I20260620 14:58:07.793408 11453 ts_manager.cc:194] Registered new tserver with Master: bf0be9d7614240ecb63b46241c237db9 (127.7.46.130:40977)
I20260620 14:58:07.794211 11453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:59403
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:58:07.812312 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34125
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:07.920598 11781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:07.920861 11781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:07.920933 11781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:07.924917 11781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:07.924997 11781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:07.925089 11781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:07.930132 11781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:07.931137 11781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:07.932132 11781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:07.941547 11789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:07.941556 11787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:07.941556 11786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:07.941771 11781 server_base.cc:1034] running on GCE node
I20260620 14:58:07.942307 11781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:07.942910 11781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:07.944054 11781 hybrid_clock.cc:648] HybridClock initialized: now 1781967487944024 us; error 45 us; skew 500 ppm
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:07.946959 11781 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:58:07.947688 11781 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:07.947750 11781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:07.948005 11781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:07.950084 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "6447d936971d47f282d1ae6acfc7654f"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "83e1d541f3ae6d6a34ab4e83fa347ba3"
server_key_iv: "9e8c005d60f0ab08186f68c801788f04"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.950636 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "6447d936971d47f282d1ae6acfc7654f"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "83e1d541f3ae6d6a34ab4e83fa347ba3"
server_key_iv: "9e8c005d60f0ab08186f68c801788f04"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.954269 11781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:58:07.956648 11795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.957862 11781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:58:07.958006 11781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "6447d936971d47f282d1ae6acfc7654f"
format_stamp: "Formatted at 2026-06-20 14:58:07 on dist-test-slave-30t4"
server_key: "83e1d541f3ae6d6a34ab4e83fa347ba3"
server_key_iv: "9e8c005d60f0ab08186f68c801788f04"
server_key_version: "encryptionkey@0"
I20260620 14:58:07.958122 11781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:07.973441 11781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:07.976395 11781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:07.976589 11781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:07.977151 11781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:07.977931 11781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:07.977977 11781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.978024 11781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:07.978080 11781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:07.987072 11781 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:36989
I20260620 14:58:07.987090 11908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:36989 every 8 connection(s)
I20260620 14:58:07.987690 11781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:07.988251 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 11781
I20260620 14:58:07.988368 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:07.988677 7354 external_mini_cluster.cc:1428] Setting key a9cbff6bd98447401e8164a9d01e5189
Jun 20 14:58:07 dist-test-slave-30t4 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967487, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:08.001266 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:07.989552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:55407 (local address 127.7.46.190:34125)
0620 14:58:07.989698 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:07.989702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:07.990669 (+ 967us) server_negotiation.cc:407] Connection header received
0620 14:58:07.992085 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:07.992088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:07.992143 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:07.992223 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:07.993371 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.993915 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.995056 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.995292 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.995403 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:07.997792 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:07.997796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:07.997798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:07.997827 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:07.999247 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:07.999749 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:07.999754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:07.999756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:07.999859 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:08.000194 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:08.000199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:08.000201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:08.000376 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:08.000492 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.000957 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.001096 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:58:08.002180 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:07.990039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34125 (local address 127.7.46.131:55407)
0620 14:58:07.990526 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:07.990568 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:07.991878 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:07.992394 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:07.992418 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:07.992767 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:07.993233 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:07.993246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.994042 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.994046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:07.994926 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:07.994935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:07.995435 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:07.995439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:07.995605 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:07.996244 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:07.996259 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:07.997637 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:07.999380 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:07.999385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:07.999398 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:07.999645 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:07.999951 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:07.999955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:07.999958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:08.000098 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:08.000481 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:08.000487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:08.000721 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 14:58:08.000902 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 14:58:08.001208 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260620 14:58:08.003176 11909 heartbeater.cc:347] Connected to a master server at 127.7.46.190:34125
I20260620 14:58:08.003410 11909 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:08.003901 11909 heartbeater.cc:511] Master 127.7.46.190:34125 requested a full tablet report, sending...
I20260620 14:58:08.004856 11453 ts_manager.cc:194] Registered new tserver with Master: 6447d936971d47f282d1ae6acfc7654f (127.7.46.131:36989)
I20260620 14:58:08.005368 11453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:55407
I20260620 14:58:08.014474 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:08.023918 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.016584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.7.46.190:34125)
0620 14:58:08.016746 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.016751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.016895 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:58:08.017052 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.017055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.017104 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.017187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:08.017866 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.018394 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.019223 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.019452 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.019567 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.020918 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:08.020924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:08.020931 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:08.020963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:08.022266 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:08.022680 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:08.022685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:08.022687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:08.022763 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:08.023029 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:08.023032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:08.023034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:08.023243 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:08.023381 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.023618 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.023734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:58:08.026588 11453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:08.029624 11453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:08.044181 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.038770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.7.46.130:40977)
0620 14:58:08.039253 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.039268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.039353 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:58:08.039490 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.039495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.039680 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.039847 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:08.040258 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.041413 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.042813 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.043463 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.043570 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.043706 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.043920 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.044019 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":117,"threads_started":1}
I20260620 14:58:08.044178 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.038340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:40977 (local address 127.0.0.1:42016)
0620 14:58:08.039272 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:08.039285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:08.039378 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:08.040024 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:08.040027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:08.040046 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:08.040135 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.040140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.041549 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.041553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.042689 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.042697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.043601 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.043607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.043713 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.043749 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:58:08.043886 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:58:08.044007 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":21,"thread_start_us":86,"threads_started":1}
I20260620 14:58:08.044637 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.038803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.7.46.129:37613)
0620 14:58:08.039167 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.039180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.039208 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:58:08.039287 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.039292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.039509 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.039689 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:08.040010 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.041071 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.042795 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.043511 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.043645 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.043816 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.044121 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.044233 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":84,"threads_started":1}
I20260620 14:58:08.044801 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.038013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:36989 (local address 127.0.0.1:39044)
0620 14:58:08.038350 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:08.038404 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:08.038542 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:08.040492 (+ 1950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:08.040495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:08.040513 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:08.040601 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.040606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.041857 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.041863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.043144 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.043156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.044220 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.044224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.044508 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.044550 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:58:08.044600 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:58:08.044676 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":50,"thread_start_us":78,"threads_started":1}
I20260620 14:58:08.044991 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.038022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:37613 (local address 127.0.0.1:56786)
0620 14:58:08.038870 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:08.038884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:08.038982 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:08.039675 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:08.039680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:08.039702 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:08.039868 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.039875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.041187 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.041193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.042665 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.042676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.043847 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.043850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.043941 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.043965 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:58:08.044777 (+ 812us) client_negotiation.cc:240] Negotiation successful
0620 14:58:08.044852 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":75,"thread_start_us":83,"threads_started":1}
I20260620 14:58:08.045534 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.038695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.7.46.131:36989)
0620 14:58:08.040032 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.040043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.040065 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:58:08.040139 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.040143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.040345 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.040493 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:08.040704 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.041736 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.043279 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.044099 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.045066 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.045223 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.045272 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.045361 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":158,"threads_started":1}
I20260620 14:58:08.047688 11710 tablet_service.cc:1467] Processing CreateTablet for tablet 63359567c9fb4496b216d8f01f98c7e0 (DEFAULT_TABLE table=test-table [id=4528fee7d27342e99bb3c7c621b6cc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:08.048197 11843 tablet_service.cc:1467] Processing CreateTablet for tablet 63359567c9fb4496b216d8f01f98c7e0 (DEFAULT_TABLE table=test-table [id=4528fee7d27342e99bb3c7c621b6cc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:08.048867 11710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63359567c9fb4496b216d8f01f98c7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:08.049168 11843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63359567c9fb4496b216d8f01f98c7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:08.053701 11577 tablet_service.cc:1467] Processing CreateTablet for tablet 63359567c9fb4496b216d8f01f98c7e0 (DEFAULT_TABLE table=test-table [id=4528fee7d27342e99bb3c7c621b6cc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:08.054695 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63359567c9fb4496b216d8f01f98c7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:08.056092 11927 tablet_bootstrap.cc:492] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Bootstrap starting.
I20260620 14:58:08.058341 11927 tablet_bootstrap.cc:654] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:08.059628 11927 log.cc:824] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:08.060006 11926 tablet_bootstrap.cc:492] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Bootstrap starting.
I20260620 14:58:08.061165 11929 tablet_bootstrap.cc:492] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9: Bootstrap starting.
I20260620 14:58:08.061859 11926 tablet_bootstrap.cc:654] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:08.063014 11929 tablet_bootstrap.cc:654] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:08.063201 11926 log.cc:824] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:08.063539 11927 tablet_bootstrap.cc:492] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: No bootstrap required, opened a new log
I20260620 14:58:08.063732 11927 ts_tablet_manager.cc:1397] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 14:58:08.064163 11929 log.cc:824] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:08.065151 11926 tablet_bootstrap.cc:492] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: No bootstrap required, opened a new log
I20260620 14:58:08.065325 11926 ts_tablet_manager.cc:1397] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:58:08.065726 11929 tablet_bootstrap.cc:492] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9: No bootstrap required, opened a new log
I20260620 14:58:08.065905 11929 ts_tablet_manager.cc:1397] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:58:08.067612 11927 raft_consensus.cc:348] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.067921 11927 raft_consensus.cc:374] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:08.067981 11927 raft_consensus.cc:729] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0be9d7614240ecb63b46241c237db9, State: Initialized, Role: FOLLOWER
I20260620 14:58:08.068445 11927 consensus_queue.cc:260] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.068969 11926 raft_consensus.cc:348] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.069216 11926 raft_consensus.cc:374] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:08.069064 11929 raft_consensus.cc:348] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.069278 11926 raft_consensus.cc:729] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6447d936971d47f282d1ae6acfc7654f, State: Initialized, Role: FOLLOWER
I20260620 14:58:08.069329 11929 raft_consensus.cc:374] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:08.069396 11929 raft_consensus.cc:729] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdfd5182908747bb98a90c3d702dd4b9, State: Initialized, Role: FOLLOWER
I20260620 14:58:08.069442 11927 ts_tablet_manager.cc:1428] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260620 14:58:08.069661 11926 consensus_queue.cc:260] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.069763 11776 heartbeater.cc:503] Master 127.7.46.190:34125 was elected leader, sending a full tablet report...
I20260620 14:58:08.070274 11909 heartbeater.cc:503] Master 127.7.46.190:34125 was elected leader, sending a full tablet report...
I20260620 14:58:08.070565 11926 ts_tablet_manager.cc:1428] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:08.071087 11929 consensus_queue.cc:260] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.071911 11929 ts_tablet_manager.cc:1428] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260620 14:58:08.072242 11643 heartbeater.cc:503] Master 127.7.46.190:34125 was elected leader, sending a full tablet report...
W20260620 14:58:08.073558 11644 tablet.cc:2369] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:08.238703 11910 tablet.cc:2369] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:08.277864 11777 tablet.cc:2369] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:08.434835 11934 raft_consensus.cc:482] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:08.435055 11934 raft_consensus.cc:504] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.436239 11934 leader_election.cc:288] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6447d936971d47f282d1ae6acfc7654f (127.7.46.131:36989), bf0be9d7614240ecb63b46241c237db9 (127.7.46.130:40977)
I20260620 14:58:08.440342 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.436605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:40975 (local address 127.7.46.130:40977)
0620 14:58:08.436757 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.436762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.437053 (+ 291us) server_negotiation.cc:407] Connection header received
0620 14:58:08.437195 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.437200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.437256 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.437335 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:08.437632 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.438188 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.439306 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.439713 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.439871 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.439924 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.440143 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.440215 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:08.440342 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.436587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:58803 (local address 127.7.46.131:36989)
0620 14:58:08.436780 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.436785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.436847 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:58:08.437060 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.437064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.437122 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.437199 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:08.437518 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.438156 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.439338 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.439734 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.439881 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.439933 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.440140 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.440209 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 14:58:08.440362 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.436536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:40977 (local address 127.7.46.129:40975)
0620 14:58:08.436923 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:08.436934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:08.437073 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:08.437359 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:08.437361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:08.437385 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:08.437497 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.437504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.438317 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.438319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.439187 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.439195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.439857 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.439861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.439974 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.440032 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 14:58:08.440126 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:58:08.440211 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":50,"thread_start_us":102,"threads_started":1}
I20260620 14:58:08.440371 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.436425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:36989 (local address 127.7.46.129:58803)
0620 14:58:08.436760 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:08.436775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:08.436900 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:08.437230 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:08.437233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:08.437250 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:08.437393 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.437402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.438257 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.438263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.439200 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:08.439209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.439880 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.439884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:08.440017 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.440035 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:58:08.440123 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:58:08.440221 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":54,"thread_start_us":89,"threads_started":1}
I20260620 14:58:08.441761 11730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63359567c9fb4496b216d8f01f98c7e0" candidate_uuid: "bdfd5182908747bb98a90c3d702dd4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf0be9d7614240ecb63b46241c237db9" is_pre_election: true
I20260620 14:58:08.441941 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63359567c9fb4496b216d8f01f98c7e0" candidate_uuid: "bdfd5182908747bb98a90c3d702dd4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6447d936971d47f282d1ae6acfc7654f" is_pre_election: true
I20260620 14:58:08.442072 11730 raft_consensus.cc:2454] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdfd5182908747bb98a90c3d702dd4b9 in term 0.
I20260620 14:58:08.442260 11863 raft_consensus.cc:2454] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdfd5182908747bb98a90c3d702dd4b9 in term 0.
I20260620 14:58:08.442575 11531 leader_election.cc:302] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdfd5182908747bb98a90c3d702dd4b9, bf0be9d7614240ecb63b46241c237db9; no voters:
I20260620 14:58:08.442873 11934 raft_consensus.cc:2781] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:08.442955 11934 raft_consensus.cc:482] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:08.442987 11934 raft_consensus.cc:3037] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:08.444025 11934 raft_consensus.cc:504] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.444392 11934 leader_election.cc:288] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [CANDIDATE]: Term 1 election: Requested vote from peers 6447d936971d47f282d1ae6acfc7654f (127.7.46.131:36989), bf0be9d7614240ecb63b46241c237db9 (127.7.46.130:40977)
I20260620 14:58:08.444756 11730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63359567c9fb4496b216d8f01f98c7e0" candidate_uuid: "bdfd5182908747bb98a90c3d702dd4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf0be9d7614240ecb63b46241c237db9"
I20260620 14:58:08.444766 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63359567c9fb4496b216d8f01f98c7e0" candidate_uuid: "bdfd5182908747bb98a90c3d702dd4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6447d936971d47f282d1ae6acfc7654f"
I20260620 14:58:08.444878 11730 raft_consensus.cc:3037] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:08.444891 11863 raft_consensus.cc:3037] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:08.445896 11730 raft_consensus.cc:2454] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdfd5182908747bb98a90c3d702dd4b9 in term 1.
I20260620 14:58:08.445946 11863 raft_consensus.cc:2454] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdfd5182908747bb98a90c3d702dd4b9 in term 1.
I20260620 14:58:08.446218 11531 leader_election.cc:302] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdfd5182908747bb98a90c3d702dd4b9, bf0be9d7614240ecb63b46241c237db9; no voters:
I20260620 14:58:08.446370 11934 raft_consensus.cc:2781] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:08.446594 11934 raft_consensus.cc:686] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [term 1 LEADER]: Becoming Leader. State: Replica: bdfd5182908747bb98a90c3d702dd4b9, State: Running, Role: LEADER
I20260620 14:58:08.446905 11934 consensus_queue.cc:237] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } }
I20260620 14:58:08.449955 11451 catalog_manager.cc:5495] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdfd5182908747bb98a90c3d702dd4b9 (127.7.46.129). New cstate: current_term: 1 leader_uuid: "bdfd5182908747bb98a90c3d702dd4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdfd5182908747bb98a90c3d702dd4b9" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 } health_report { overall_health: UNKNOWN } } }
I20260620 14:58:08.493191 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.489665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.7.46.129:37613)
0620 14:58:08.489811 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.489817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.489832 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:58:08.489960 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.489975 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.490029 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.490140 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:08.490388 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.490888 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.491731 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:08.491978 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:08.492090 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:08.492398 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:08.492798 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:08.492929 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.492986 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.493041 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:58:08.498696 11863 raft_consensus.cc:1261] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 1 FOLLOWER]: Refusing update from remote peer bdfd5182908747bb98a90c3d702dd4b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:08.498701 11730 raft_consensus.cc:1261] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 1 FOLLOWER]: Refusing update from remote peer bdfd5182908747bb98a90c3d702dd4b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:08.499402 11937 consensus_queue.cc:1035] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0be9d7614240ecb63b46241c237db9" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 40977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:08.499601 11934 consensus_queue.cc:1035] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6447d936971d47f282d1ae6acfc7654f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:08.505877 11940 mvcc.cc:205] Tried to move back new op lower bound from 7298938832883875840 to 7298938832680968192. Current Snapshot: MvccSnapshot[applied={T|T < 7298938832883875840}]
I20260620 14:58:08.506688 11941 mvcc.cc:205] Tried to move back new op lower bound from 7298938832883875840 to 7298938832680968192. Current Snapshot: MvccSnapshot[applied={T|T < 7298938832883875840}]
I20260620 14:58:08.507920 11942 mvcc.cc:205] Tried to move back new op lower bound from 7298938832883875840 to 7298938832680968192. Current Snapshot: MvccSnapshot[applied={T|T < 7298938832883875840}]
I20260620 14:58:08.514403 11453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:08.514585 11453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:08.516930 11453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28b4b7f086ba4803b0f209d8e0a08011: Sending DeleteTablet for 3 replicas of tablet 63359567c9fb4496b216d8f01f98c7e0
I20260620 14:58:08.517635 11577 tablet_service.cc:1514] Processing DeleteTablet for tablet 63359567c9fb4496b216d8f01f98c7e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:08 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:56786
I20260620 14:58:08.517668 11843 tablet_service.cc:1514] Processing DeleteTablet for tablet 63359567c9fb4496b216d8f01f98c7e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:08 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:39044
I20260620 14:58:08.517755 11710 tablet_service.cc:1514] Processing DeleteTablet for tablet 63359567c9fb4496b216d8f01f98c7e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:08 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:42016
I20260620 14:58:08.518153 11951 tablet_replica.cc:330] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: stopping tablet replica
I20260620 14:58:08.518381 11949 tablet_replica.cc:330] T 63359567c9fb4496b216d8f01f98c7e0 P bdfd5182908747bb98a90c3d702dd4b9: stopping tablet replica
I20260620 14:58:08.518446 11951 raft_consensus.cc:2229] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:08.518726 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 11515
I20260620 14:58:08.518771 11951 raft_consensus.cc:2258] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:08.520119 11950 tablet_replica.cc:330] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: stopping tablet replica
I20260620 14:58:08.520502 11950 raft_consensus.cc:2229] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:08.520808 11950 raft_consensus.cc:2258] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:08.520813 11951 ts_tablet_manager.cc:1905] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:08.521929 11950 ts_tablet_manager.cc:1905] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:08.524057 11950 ts_tablet_manager.cc:1918] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:08.524187 11950 log.cc:1196] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/63359567c9fb4496b216d8f01f98c7e0
I20260620 14:58:08.524240 11951 ts_tablet_manager.cc:1918] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:08.524330 11951 log.cc:1196] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/63359567c9fb4496b216d8f01f98c7e0
I20260620 14:58:08.524497 11950 ts_tablet_manager.cc:1939] T 63359567c9fb4496b216d8f01f98c7e0 P 6447d936971d47f282d1ae6acfc7654f: Deleting consensus metadata
I20260620 14:58:08.524642 11951 ts_tablet_manager.cc:1939] T 63359567c9fb4496b216d8f01f98c7e0 P bf0be9d7614240ecb63b46241c237db9: Deleting consensus metadata
I20260620 14:58:08.525666 11437 catalog_manager.cc:4841] TS 6447d936971d47f282d1ae6acfc7654f (127.7.46.131:36989): tablet 63359567c9fb4496b216d8f01f98c7e0 (table test-table [id=4528fee7d27342e99bb3c7c621b6cc63]) successfully deleted
I20260620 14:58:08.525897 11438 catalog_manager.cc:4841] TS bf0be9d7614240ecb63b46241c237db9 (127.7.46.130:40977): tablet 63359567c9fb4496b216d8f01f98c7e0 (table test-table [id=4528fee7d27342e99bb3c7c621b6cc63]) successfully deleted
W20260620 14:58:08.526520 11439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:37613 (error 108)
I20260620 14:58:08.526672 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 11648
I20260620 14:58:08.527640 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.527282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:37613 (local address 127.0.0.1:56798)
0620 14:58:08.527472 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:08.527545 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 14:58:08.527829 11439 catalog_manager.cc:4584] TS bdfd5182908747bb98a90c3d702dd4b9 (127.7.46.129:37613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63359567c9fb4496b216d8f01f98c7e0: Network error: Client connection negotiation failed: client connection to 127.7.46.129:37613: connect: Connection refused (error 111)
I20260620 14:58:08.531977 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 11781
I20260620 14:58:08.537386 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 11422
2026-06-20T14:58:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2570 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:08Z Disabled control of system clock
I20260620 14:58:08.566701 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:40403
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:40403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:58:08.676170 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:08.676414 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:08.676450 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:08.680332 11956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:08.680380 11956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:08.680397 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:08.680413 11956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:08.680428 11956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:08.685138 11956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:40403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:40403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:08.686075 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:08.687321 11956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:08.695536 11962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:08.695545 11961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:08.695593 11964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:08.695689 11956 server_base.cc:1034] running on GCE node
I20260620 14:58:08.696362 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:08.697358 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:08.698520 11956 hybrid_clock.cc:648] HybridClock initialized: now 1781967488698493 us; error 41 us; skew 500 ppm
I20260620 14:58:08.700273 11956 webserver.cc:460] Webserver started at http://127.7.46.190:45605/ using document root <none> and password file <none>
I20260620 14:58:08.700794 11956 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:08.700853 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:08.701005 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:08.702986 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "e2ccacac487541df9af87baa5e36fc13"
format_stamp: "Formatted at 2026-06-20 14:58:08 on dist-test-slave-30t4"
server_key: "9459e7b6c81619f23692bbb75e6d0d3f"
server_key_iv: "ff8a101312e5ed5988bfb489f7055ea5"
server_key_version: "encryptionkey@0"
I20260620 14:58:08.703526 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "e2ccacac487541df9af87baa5e36fc13"
format_stamp: "Formatted at 2026-06-20 14:58:08 on dist-test-slave-30t4"
server_key: "9459e7b6c81619f23692bbb75e6d0d3f"
server_key_iv: "ff8a101312e5ed5988bfb489f7055ea5"
server_key_version: "encryptionkey@0"
I20260620 14:58:08.707067 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:58:08.709363 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:08.710536 11956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:08.710683 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "e2ccacac487541df9af87baa5e36fc13"
format_stamp: "Formatted at 2026-06-20 14:58:08 on dist-test-slave-30t4"
server_key: "9459e7b6c81619f23692bbb75e6d0d3f"
server_key_iv: "ff8a101312e5ed5988bfb489f7055ea5"
server_key_version: "encryptionkey@0"
I20260620 14:58:08.710796 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:08.715194 11956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:08.715818 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:08.715998 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:08.722613 11956 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:40403
I20260620 14:58:08.722637 12022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:40403 every 8 connection(s)
I20260620 14:58:08.723466 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:08.726472 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:08.731614 12023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13: Bootstrap starting.
I20260620 14:58:08.732133 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 11956
I20260620 14:58:08.732244 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:08.732540 7354 external_mini_cluster.cc:1428] Setting key be73cd9ce23c33d81cb8919d74472715
I20260620 14:58:08.734403 12023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:08.735203 12023 log.cc:824] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:08.737138 12023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13: No bootstrap required, opened a new log
I20260620 14:58:08.738083 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.733889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.7.46.190:40403)
0620 14:58:08.734526 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.734542 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.734586 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:58:08.735240 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.735256 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.735705 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.736114 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:08.736897 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:08.736902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:08.736912 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:08.736922 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:08.736975 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:08.736991 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:08.737105 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.737200 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.737438 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260620 14:58:08.741005 12023 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ccacac487541df9af87baa5e36fc13" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40403 } }
I20260620 14:58:08.741245 12023 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:08.741322 12023 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ccacac487541df9af87baa5e36fc13, State: Initialized, Role: FOLLOWER
I20260620 14:58:08.741719 12023 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ccacac487541df9af87baa5e36fc13" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40403 } }
I20260620 14:58:08.741874 12023 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:08.741958 12023 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:08.742082 12023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:08.742966 12023 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ccacac487541df9af87baa5e36fc13" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40403 } }
I20260620 14:58:08.743294 12023 leader_election.cc:302] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2ccacac487541df9af87baa5e36fc13; no voters:
I20260620 14:58:08.743613 12023 leader_election.cc:288] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:08.743721 12028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:08.743948 12028 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [term 1 LEADER]: Becoming Leader. State: Replica: e2ccacac487541df9af87baa5e36fc13, State: Running, Role: LEADER
I20260620 14:58:08.744259 12028 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ccacac487541df9af87baa5e36fc13" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40403 } }
I20260620 14:58:08.744813 12023 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:08.745366 12028 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ccacac487541df9af87baa5e36fc13. Latest consensus state: current_term: 1 leader_uuid: "e2ccacac487541df9af87baa5e36fc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ccacac487541df9af87baa5e36fc13" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40403 } } }
I20260620 14:58:08.745481 12028 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:08.746151 12029 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2ccacac487541df9af87baa5e36fc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ccacac487541df9af87baa5e36fc13" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40403 } } }
I20260620 14:58:08.746270 12029 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:08.746618 12033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:08.749754 12033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:08.755626 12033 catalog_manager.cc:1347] Generated new cluster ID: 5caa8ac40c8546f78fbc38c9e193046c
I20260620 14:58:08.755692 12033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:08.764420 12033 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:08.764734 12033 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:08.771577 12033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13: Generated new TSK 0
I20260620 14:58:08.772208 12033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:08.774603 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40403
--builtin_ntp_servers=127.7.46.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:58:08.904577 12047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:08.904811 12047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:08.904851 12047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:08.908896 12047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:08.908950 12047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:08.909035 12047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:08.913905 12047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:08.915064 12047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:08.916455 12047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:08.926128 12053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:08.926187 12052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:08.926240 12047 server_base.cc:1034] running on GCE node
W20260620 14:58:08.926158 12055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:08.926851 12047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:08.927587 12047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:08.928869 12047 hybrid_clock.cc:648] HybridClock initialized: now 1781967488928788 us; error 93 us; skew 500 ppm
I20260620 14:58:08.931272 12047 webserver.cc:460] Webserver started at http://127.7.46.129:46373/ using document root <none> and password file <none>
I20260620 14:58:08.932104 12047 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:08.932206 12047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:08.932459 12047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:08.935364 12047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "6834a199661f4afd9364b02f576db859"
format_stamp: "Formatted at 2026-06-20 14:58:08 on dist-test-slave-30t4"
server_key: "d3ad23f597bee0cb789910ea26397598"
server_key_iv: "0a9a02cdfa358e296942815e194f312a"
server_key_version: "encryptionkey@0"
I20260620 14:58:08.936008 12047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "6834a199661f4afd9364b02f576db859"
format_stamp: "Formatted at 2026-06-20 14:58:08 on dist-test-slave-30t4"
server_key: "d3ad23f597bee0cb789910ea26397598"
server_key_iv: "0a9a02cdfa358e296942815e194f312a"
server_key_version: "encryptionkey@0"
I20260620 14:58:08.941099 12047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:58:08.944329 12061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:08.945900 12047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:08.946061 12047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "6834a199661f4afd9364b02f576db859"
format_stamp: "Formatted at 2026-06-20 14:58:08 on dist-test-slave-30t4"
server_key: "d3ad23f597bee0cb789910ea26397598"
server_key_iv: "0a9a02cdfa358e296942815e194f312a"
server_key_version: "encryptionkey@0"
I20260620 14:58:08.946193 12047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:08.951705 12047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:08.952400 12047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:08.952598 12047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:08.953289 12047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:08.954587 12047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:08.954646 12047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:08.954715 12047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:08.954746 12047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:08.965926 12047 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:34125
I20260620 14:58:08.965991 12174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:34125 every 8 connection(s)
I20260620 14:58:08.966869 12047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:08.971244 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 12047
I20260620 14:58:08.971382 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:08.971676 7354 external_mini_cluster.cc:1428] Setting key f98709dfbd94cae152b33ac00c135fb2
I20260620 14:58:08.973960 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40403
--builtin_ntp_servers=127.7.46.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:58:08.975754 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.969136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:50117 (local address 127.7.46.190:40403)
0620 14:58:08.969311 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:08.969325 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:08.970829 (+ 1504us) server_negotiation.cc:407] Connection header received
0620 14:58:08.973511 (+ 2682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:08.973515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:08.973556 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:08.973655 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:08.975134 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:08.975142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:08.975148 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:08.975166 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:08.975197 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:08.975213 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:08.975319 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:08.975560 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:58:08.975609 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:58:08.977123 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:08.969718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40403 (local address 127.7.46.129:50117)
0620 14:58:08.970606 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:08.970665 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:08.973247 (+ 2582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:08.973849 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:08.973861 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:08.973991 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:08.974773 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:08.974776 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:08.974840 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:08.974941 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:08.975376 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:08.975390 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:08.975401 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:58:08.975592 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:58:08.976150 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":147,"threads_started":1}
I20260620 14:58:08.978375 12175 heartbeater.cc:347] Connected to a master server at 127.7.46.190:40403
I20260620 14:58:08.978634 12175 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:08.979040 12175 heartbeater.cc:511] Master 127.7.46.190:40403 requested a full tablet report, sending...
I20260620 14:58:08.980561 11987 ts_manager.cc:194] Registered new tserver with Master: 6834a199661f4afd9364b02f576db859 (127.7.46.129:34125)
W20260620 14:58:09.086443 12178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:09.086699 12178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:09.086740 12178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:09.090699 12178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:09.090756 12178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:09.090914 12178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:09.095726 12178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:09.096850 12178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:09.098189 12178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:09.107736 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.107788 12186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.107738 12184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:09.107924 12178 server_base.cc:1034] running on GCE node
I20260620 14:58:09.108466 12178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:09.109004 12178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:09.110179 12178 hybrid_clock.cc:648] HybridClock initialized: now 1781967489110165 us; error 23 us; skew 500 ppm
I20260620 14:58:09.111995 12178 webserver.cc:460] Webserver started at http://127.7.46.130:38895/ using document root <none> and password file <none>
I20260620 14:58:09.112500 12178 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:09.112555 12178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:09.112710 12178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:09.114732 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "e44ef6b549db4d72a7940a890d75abea"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "25d0e731c7ee61ff8b9b1e3d2913ced0"
server_key_iv: "d1836023f2282f5fb9154e96a3021bb5"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.115273 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "e44ef6b549db4d72a7940a890d75abea"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "25d0e731c7ee61ff8b9b1e3d2913ced0"
server_key_iv: "d1836023f2282f5fb9154e96a3021bb5"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.118667 12178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:58:09.120973 12192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.122279 12178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:09.122372 12178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "e44ef6b549db4d72a7940a890d75abea"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "25d0e731c7ee61ff8b9b1e3d2913ced0"
server_key_iv: "d1836023f2282f5fb9154e96a3021bb5"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.122454 12178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:09.126773 12178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:09.127280 12178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:09.127403 12178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:09.127902 12178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:09.128683 12178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:09.128726 12178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.128762 12178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:09.128783 12178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.137153 12178 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:36977
I20260620 14:58:09.137188 12305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:36977 every 8 connection(s)
I20260620 14:58:09.137979 12178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:09.141788 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 12178
I20260620 14:58:09.141885 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:09.142174 7354 external_mini_cluster.cc:1428] Setting key 0ffacd1bedc44bd5a1b134170339e4fa
I20260620 14:58:09.143970 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.139643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:56529 (local address 127.7.46.190:40403)
0620 14:58:09.139830 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.139835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.140879 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 14:58:09.142253 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.142256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.142301 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.142391 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.143464 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.143469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.143472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.143484 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.143509 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.143520 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.143599 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.143795 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.143840 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:58:09.144328 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40403
--builtin_ntp_servers=127.7.46.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:58:09.145059 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.140066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40403 (local address 127.7.46.130:56529)
0620 14:58:09.140726 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.140765 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:09.142048 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:09.142552 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:09.142558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:09.142644 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:09.143176 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:09.143178 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:09.143227 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:09.143304 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:09.143614 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:09.143622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:09.143628 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:58:09.143850 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 14:58:09.144156 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260620 14:58:09.146332 12306 heartbeater.cc:347] Connected to a master server at 127.7.46.190:40403
I20260620 14:58:09.146567 12306 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:09.147018 12306 heartbeater.cc:511] Master 127.7.46.190:40403 requested a full tablet report, sending...
I20260620 14:58:09.148029 11987 ts_manager.cc:194] Registered new tserver with Master: e44ef6b549db4d72a7940a890d75abea (127.7.46.130:36977)
W20260620 14:58:09.255139 12309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:09.255411 12309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:09.255491 12309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:09.259454 12309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:09.259531 12309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:09.259614 12309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:09.264339 12309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:09.265348 12309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:09.266664 12309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:09.276095 12317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.276103 12314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:09.276268 12309 server_base.cc:1034] running on GCE node
W20260620 14:58:09.276096 12315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:09.276806 12309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:09.277442 12309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:09.278620 12309 hybrid_clock.cc:648] HybridClock initialized: now 1781967489278599 us; error 33 us; skew 500 ppm
I20260620 14:58:09.280351 12309 webserver.cc:460] Webserver started at http://127.7.46.131:41107/ using document root <none> and password file <none>
I20260620 14:58:09.280900 12309 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:09.280987 12309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:09.281206 12309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:09.283159 12309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "c9d2d82378214c5685f4c8ef11b39506"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "3de37335f74b663a3a39b440b7547aee"
server_key_iv: "1c88eeaf94d6a5393ee1497d8bd5f6c1"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.283708 12309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "c9d2d82378214c5685f4c8ef11b39506"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "3de37335f74b663a3a39b440b7547aee"
server_key_iv: "1c88eeaf94d6a5393ee1497d8bd5f6c1"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.287256 12309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:58:09.289619 12323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.290807 12309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:09.290930 12309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "c9d2d82378214c5685f4c8ef11b39506"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "3de37335f74b663a3a39b440b7547aee"
server_key_iv: "1c88eeaf94d6a5393ee1497d8bd5f6c1"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.291033 12309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:09.295370 12309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:09.295984 12309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:09.296150 12309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:09.296656 12309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:09.297494 12309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:09.297569 12309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.297632 12309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:09.297679 12309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.306176 12309 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:45139
I20260620 14:58:09.306198 12436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:45139 every 8 connection(s)
I20260620 14:58:09.307055 12309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:09.310657 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 12309
I20260620 14:58:09.310779 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:09.311098 7354 external_mini_cluster.cc:1428] Setting key 17c9591fdd614c1010139e6a9d7e50c4
I20260620 14:58:09.313265 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.308807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:34177 (local address 127.7.46.190:40403)
0620 14:58:09.308937 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.308940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.310039 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 14:58:09.311451 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.311455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.311500 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.311605 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.312805 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.312809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.312812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.312823 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.312845 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.312853 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.312953 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.313099 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.313141 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:58:09.314116 12439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.309131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40403 (local address 127.7.46.131:34177)
0620 14:58:09.309780 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.309818 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:09.311246 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:09.311802 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:09.311815 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:09.311917 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:09.312552 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:09.312553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:09.312593 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:09.312656 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:09.312943 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:09.312951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:09.312957 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:58:09.313140 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 14:58:09.313369 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260620 14:58:09.315157 12437 heartbeater.cc:347] Connected to a master server at 127.7.46.190:40403
I20260620 14:58:09.315353 12437 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:09.315690 12437 heartbeater.cc:511] Master 127.7.46.190:40403 requested a full tablet report, sending...
I20260620 14:58:09.316642 11987 ts_manager.cc:194] Registered new tserver with Master: c9d2d82378214c5685f4c8ef11b39506 (127.7.46.131:45139)
I20260620 14:58:09.324743 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:09.328150 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.326778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.7.46.190:40403)
0620 14:58:09.326893 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.326896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.327081 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:58:09.327168 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.327170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.327200 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.327264 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.327881 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.327884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.327886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.327895 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.327911 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.327918 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.327978 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.328026 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.328056 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:58:09.330590 11987 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:09.333633 11987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:09.344914 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.341481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:45139 (local address 127.0.0.1:37422)
0620 14:58:09.342285 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.342332 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:09.342436 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:09.343615 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:09.343618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:09.343632 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:09.344251 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:09.344254 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:09.344265 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:09.344297 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:09.344592 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:09.344600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:09.344605 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:58:09.344708 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:58:09.344772 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":59,"threads_started":1}
I20260620 14:58:09.344930 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.341641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.7.46.131:45139)
0620 14:58:09.342081 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.342094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.342549 (+ 455us) server_negotiation.cc:407] Connection header received
0620 14:58:09.342607 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.342613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.342810 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.342943 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.344419 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.344425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.344438 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.344449 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.344492 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.344523 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.344621 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.344720 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.344788 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":132,"threads_started":1}
I20260620 14:58:09.345163 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.342182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:34125 (local address 127.0.0.1:37536)
0620 14:58:09.342450 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.342466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:09.342550 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:09.343350 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:09.343356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:09.343467 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:09.344078 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:09.344080 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:09.344101 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:09.344157 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:09.344950 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:09.344952 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:09.344955 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:58:09.345030 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:58:09.345059 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260620 14:58:09.345527 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.342345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.7.46.129:34125)
0620 14:58:09.342974 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.342987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.343022 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:58:09.343073 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.343080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.343207 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.343346 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.344688 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.344692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.344701 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.344714 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.344752 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.344821 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.344914 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.345298 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.345364 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":131,"threads_started":1}
I20260620 14:58:09.345537 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.341852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.7.46.130:36977)
0620 14:58:09.342772 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.342803 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.342913 (+ 110us) server_negotiation.cc:407] Connection header received
0620 14:58:09.343078 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.343084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.343258 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.343370 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.344582 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.344587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.344597 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.344609 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.344654 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.344684 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.344790 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.345298 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.345364 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":143,"threads_started":1}
I20260620 14:58:09.345784 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.341704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:36977 (local address 127.0.0.1:46208)
0620 14:58:09.342819 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.342832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:09.342964 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:09.343905 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:09.343910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:09.343925 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:09.344416 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:09.344418 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:09.344427 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:09.344449 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:09.344832 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:09.344836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:09.344841 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:58:09.344914 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:58:09.344952 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"spinlock_wait_cycles":6144,"thread_start_us":84,"threads_started":1}
I20260620 14:58:09.348254 12240 tablet_service.cc:1467] Processing CreateTablet for tablet 8cfc0004904943f4aa6089e8cf376073 (DEFAULT_TABLE table=test-table [id=02b153f7a0d8474395fbaf5550fed057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:09.348280 12371 tablet_service.cc:1467] Processing CreateTablet for tablet 8cfc0004904943f4aa6089e8cf376073 (DEFAULT_TABLE table=test-table [id=02b153f7a0d8474395fbaf5550fed057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:09.348279 12109 tablet_service.cc:1467] Processing CreateTablet for tablet 8cfc0004904943f4aa6089e8cf376073 (DEFAULT_TABLE table=test-table [id=02b153f7a0d8474395fbaf5550fed057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:09.349221 12240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cfc0004904943f4aa6089e8cf376073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:09.349251 12109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cfc0004904943f4aa6089e8cf376073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:09.349251 12371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cfc0004904943f4aa6089e8cf376073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:09.355938 12454 tablet_bootstrap.cc:492] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Bootstrap starting.
I20260620 14:58:09.356709 12456 tablet_bootstrap.cc:492] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859: Bootstrap starting.
I20260620 14:58:09.356822 12455 tablet_bootstrap.cc:492] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: Bootstrap starting.
I20260620 14:58:09.358222 12454 tablet_bootstrap.cc:654] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:09.358486 12456 tablet_bootstrap.cc:654] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:09.358752 12455 tablet_bootstrap.cc:654] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:09.359086 12454 log.cc:824] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:09.359242 12456 log.cc:824] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:09.359503 12455 log.cc:824] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:09.361043 12454 tablet_bootstrap.cc:492] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: No bootstrap required, opened a new log
I20260620 14:58:09.361138 12456 tablet_bootstrap.cc:492] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859: No bootstrap required, opened a new log
I20260620 14:58:09.361263 12454 ts_tablet_manager.cc:1397] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:58:09.361315 12456 ts_tablet_manager.cc:1397] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:58:09.361382 12455 tablet_bootstrap.cc:492] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: No bootstrap required, opened a new log
I20260620 14:58:09.361549 12455 ts_tablet_manager.cc:1397] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:58:09.364742 12454 raft_consensus.cc:348] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.365002 12454 raft_consensus.cc:374] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:09.365072 12454 raft_consensus.cc:729] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44ef6b549db4d72a7940a890d75abea, State: Initialized, Role: FOLLOWER
I20260620 14:58:09.365728 12456 raft_consensus.cc:348] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.365728 12455 raft_consensus.cc:348] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.365962 12455 raft_consensus.cc:374] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:09.365962 12456 raft_consensus.cc:374] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:09.366019 12455 raft_consensus.cc:729] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d2d82378214c5685f4c8ef11b39506, State: Initialized, Role: FOLLOWER
I20260620 14:58:09.366031 12456 raft_consensus.cc:729] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6834a199661f4afd9364b02f576db859, State: Initialized, Role: FOLLOWER
I20260620 14:58:09.366462 12456 consensus_queue.cc:260] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.366462 12455 consensus_queue.cc:260] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.366873 12454 consensus_queue.cc:260] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.367354 12437 heartbeater.cc:503] Master 127.7.46.190:40403 was elected leader, sending a full tablet report...
I20260620 14:58:09.367460 12456 ts_tablet_manager.cc:1428] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260620 14:58:09.367460 12455 ts_tablet_manager.cc:1428] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 14:58:09.367763 12175 heartbeater.cc:503] Master 127.7.46.190:40403 was elected leader, sending a full tablet report...
I20260620 14:58:09.368247 12306 heartbeater.cc:503] Master 127.7.46.190:40403 was elected leader, sending a full tablet report...
I20260620 14:58:09.368417 12454 ts_tablet_manager.cc:1428] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:58:09.386761 12461 raft_consensus.cc:482] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:09.386929 12461 raft_consensus.cc:504] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.387969 12461 leader_election.cc:288] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9d2d82378214c5685f4c8ef11b39506 (127.7.46.131:45139), e44ef6b549db4d72a7940a890d75abea (127.7.46.130:36977)
W20260620 14:58:09.388872 12307 tablet.cc:2369] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:09.389760 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.388178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:45139 (local address 127.7.46.129:45267)
0620 14:58:09.388298 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.388315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:09.388386 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:09.388661 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:09.388685 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:09.388702 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:09.389223 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:09.389226 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:09.389237 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:09.389270 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:09.389519 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:09.389522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:09.389525 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:58:09.389602 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:58:09.389637 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:58:09.389760 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.388261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:45267 (local address 127.7.46.131:45139)
0620 14:58:09.388380 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.388385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.388401 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:09.388496 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.388500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.388538 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.388629 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.389394 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.389397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.389400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.389412 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.389432 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.389443 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.389529 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.389599 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.389637 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:58:09.390184 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.388337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:40409 (local address 127.7.46.130:36977)
0620 14:58:09.388473 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.388477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.388752 (+ 275us) server_negotiation.cc:407] Connection header received
0620 14:58:09.388854 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.388856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.388909 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.388985 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.389700 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.389703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.389706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.389718 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.389737 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.389748 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.389850 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.390033 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.390075 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:58:09.390205 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.388194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:36977 (local address 127.7.46.129:40409)
0620 14:58:09.388674 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.388686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:09.388767 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:09.389074 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:09.389077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:09.389091 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:09.389539 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:09.389541 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:09.389550 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:09.389581 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:09.389943 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:09.389946 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:09.389949 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:58:09.390025 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:58:09.390062 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":118,"thread_start_us":150,"threads_started":1}
I20260620 14:58:09.390338 12391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cfc0004904943f4aa6089e8cf376073" candidate_uuid: "6834a199661f4afd9364b02f576db859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d2d82378214c5685f4c8ef11b39506" is_pre_election: true
I20260620 14:58:09.390616 12391 raft_consensus.cc:2454] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6834a199661f4afd9364b02f576db859 in term 0.
I20260620 14:58:09.390856 12260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cfc0004904943f4aa6089e8cf376073" candidate_uuid: "6834a199661f4afd9364b02f576db859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44ef6b549db4d72a7940a890d75abea" is_pre_election: true
I20260620 14:58:09.391080 12065 leader_election.cc:302] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6834a199661f4afd9364b02f576db859, c9d2d82378214c5685f4c8ef11b39506; no voters:
I20260620 14:58:09.391125 12260 raft_consensus.cc:2454] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6834a199661f4afd9364b02f576db859 in term 0.
I20260620 14:58:09.391368 12461 raft_consensus.cc:2781] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:09.391469 12461 raft_consensus.cc:482] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:09.391533 12461 raft_consensus.cc:3037] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:09.392529 12461 raft_consensus.cc:504] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.392875 12461 leader_election.cc:288] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [CANDIDATE]: Term 1 election: Requested vote from peers c9d2d82378214c5685f4c8ef11b39506 (127.7.46.131:45139), e44ef6b549db4d72a7940a890d75abea (127.7.46.130:36977)
I20260620 14:58:09.393189 12391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cfc0004904943f4aa6089e8cf376073" candidate_uuid: "6834a199661f4afd9364b02f576db859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d2d82378214c5685f4c8ef11b39506"
I20260620 14:58:09.393203 12260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cfc0004904943f4aa6089e8cf376073" candidate_uuid: "6834a199661f4afd9364b02f576db859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44ef6b549db4d72a7940a890d75abea"
I20260620 14:58:09.393297 12391 raft_consensus.cc:3037] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:09.393321 12260 raft_consensus.cc:3037] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:09.394279 12391 raft_consensus.cc:2454] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6834a199661f4afd9364b02f576db859 in term 1.
I20260620 14:58:09.394526 12260 raft_consensus.cc:2454] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6834a199661f4afd9364b02f576db859 in term 1.
I20260620 14:58:09.394577 12065 leader_election.cc:302] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6834a199661f4afd9364b02f576db859, c9d2d82378214c5685f4c8ef11b39506; no voters:
I20260620 14:58:09.394752 12461 raft_consensus.cc:2781] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:09.394961 12461 raft_consensus.cc:686] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [term 1 LEADER]: Becoming Leader. State: Replica: 6834a199661f4afd9364b02f576db859, State: Running, Role: LEADER
I20260620 14:58:09.395273 12461 consensus_queue.cc:237] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } }
I20260620 14:58:09.398495 11985 catalog_manager.cc:5495] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6834a199661f4afd9364b02f576db859 (127.7.46.129). New cstate: current_term: 1 leader_uuid: "6834a199661f4afd9364b02f576db859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6834a199661f4afd9364b02f576db859" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 } health_report { overall_health: UNKNOWN } } }
I20260620 14:58:09.414681 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.413060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.7.46.129:34125)
0620 14:58:09.413177 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.413182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.413226 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:58:09.413430 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.413433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.413485 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.413573 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.414294 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.414298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.414301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.414311 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.414331 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.414342 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.414439 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.414507 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.414545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:58:09.420257 12391 raft_consensus.cc:1261] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 1 FOLLOWER]: Refusing update from remote peer 6834a199661f4afd9364b02f576db859: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:09.420256 12260 raft_consensus.cc:1261] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 1 FOLLOWER]: Refusing update from remote peer 6834a199661f4afd9364b02f576db859: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:09.420949 12461 consensus_queue.cc:1035] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e44ef6b549db4d72a7940a890d75abea" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:09.421144 12464 consensus_queue.cc:1035] T 8cfc0004904943f4aa6089e8cf376073 P 6834a199661f4afd9364b02f576db859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9d2d82378214c5685f4c8ef11b39506" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:09.429455 12468 mvcc.cc:205] Tried to move back new op lower bound from 7298938836658180096 to 7298938836565807104. Current Snapshot: MvccSnapshot[applied={T|T < 7298938836658180096}]
I20260620 14:58:09.430066 12469 mvcc.cc:205] Tried to move back new op lower bound from 7298938836658180096 to 7298938836565807104. Current Snapshot: MvccSnapshot[applied={T|T < 7298938836658180096}]
I20260620 14:58:09.436058 11987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:09.436193 11987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:09.438488 11987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2ccacac487541df9af87baa5e36fc13: Sending DeleteTablet for 3 replicas of tablet 8cfc0004904943f4aa6089e8cf376073
I20260620 14:58:09.439268 12371 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cfc0004904943f4aa6089e8cf376073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:09 UTC) from {username='slave'} at 127.0.0.1:37422
I20260620 14:58:09.439424 12109 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cfc0004904943f4aa6089e8cf376073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:09 UTC) from {username='slave'} at 127.0.0.1:37536
I20260620 14:58:09.439791 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 12047
I20260620 14:58:09.439736 12240 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cfc0004904943f4aa6089e8cf376073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:09 UTC) from {username='slave'} at 127.0.0.1:46208
I20260620 14:58:09.440527 12476 tablet_replica.cc:330] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: stopping tablet replica
I20260620 14:58:09.440841 12476 raft_consensus.cc:2229] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:09.441334 12476 raft_consensus.cc:2258] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:09.442232 12478 tablet_replica.cc:330] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: stopping tablet replica
I20260620 14:58:09.442597 12478 raft_consensus.cc:2229] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:09.442914 12478 raft_consensus.cc:2258] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:09.443040 12476 ts_tablet_manager.cc:1905] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:09.445454 12478 ts_tablet_manager.cc:1905] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:09.445991 12476 ts_tablet_manager.cc:1918] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:09.446096 12476 log.cc:1196] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/8cfc0004904943f4aa6089e8cf376073
I20260620 14:58:09.446399 12476 ts_tablet_manager.cc:1939] T 8cfc0004904943f4aa6089e8cf376073 P c9d2d82378214c5685f4c8ef11b39506: Deleting consensus metadata
I20260620 14:58:09.447592 11974 catalog_manager.cc:4841] TS c9d2d82378214c5685f4c8ef11b39506 (127.7.46.131:45139): tablet 8cfc0004904943f4aa6089e8cf376073 (table test-table [id=02b153f7a0d8474395fbaf5550fed057]) successfully deleted
I20260620 14:58:09.448076 12478 ts_tablet_manager.cc:1918] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:09.448196 12478 log.cc:1196] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/8cfc0004904943f4aa6089e8cf376073
I20260620 14:58:09.448491 12478 ts_tablet_manager.cc:1939] T 8cfc0004904943f4aa6089e8cf376073 P e44ef6b549db4d72a7940a890d75abea: Deleting consensus metadata
W20260620 14:58:09.448596 11974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:34125 (error 108)
I20260620 14:58:09.448701 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 12178
I20260620 14:58:09.450264 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.449966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:34125 (local address 127.0.0.1:37548)
0620 14:58:09.450110 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.450181 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:34125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:58:09.450405 11974 catalog_manager.cc:4584] TS 6834a199661f4afd9364b02f576db859 (127.7.46.129:34125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cfc0004904943f4aa6089e8cf376073: Network error: Client connection negotiation failed: client connection to 127.7.46.129:34125: connect: Connection refused (error 111)
I20260620 14:58:09.454494 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 12309
I20260620 14:58:09.455389 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.454612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:36977 (local address 127.0.0.1:46220)
0620 14:58:09.455232 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.455295 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.130:36977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
I20260620 14:58:09.460146 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 11956
2026-06-20T14:58:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (921 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:58:09.481400 7354 mini_oidc.cc:152] Starting JWKS server
I20260620 14:58:09.482955 7354 webserver.cc:460] Webserver started at https://127.0.0.1:32965/ using document root <none> and password file <none>
I20260620 14:58:09.483147 7354 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:58:09.483418 7354 webserver.cc:460] Webserver started at http://127.0.0.1:34771/ using document root <none> and password file <none>
2026-06-20T14:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:09Z Disabled control of system clock
I20260620 14:58:09.492301 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:45435
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:45435
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32965/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/testchainca.cert with env {}
W20260620 14:58:09.598824 12485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:09.599061 12485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:09.599097 12485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:09.602924 12485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:09.602972 12485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:09.602988 12485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:09.603004 12485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:09.603024 12485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:58:09.603039 12485 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:58:09.603053 12485 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32965/jwks.json
I20260620 14:58:09.608348 12485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:45435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32965/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:09.609237 12485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:09.610142 12485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:09.618188 12491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.618161 12490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.618156 12493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:09.618229 12485 server_base.cc:1034] running on GCE node
I20260620 14:58:09.618868 12485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:09.619921 12485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:09.621078 12485 hybrid_clock.cc:648] HybridClock initialized: now 1781967489621061 us; error 29 us; skew 500 ppm
I20260620 14:58:09.622769 12485 webserver.cc:460] Webserver started at http://127.7.46.190:40049/ using document root <none> and password file <none>
I20260620 14:58:09.623299 12485 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:09.623364 12485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:09.623585 12485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:09.625576 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "aa6e8b5eeba74eefa4eb7defaeada375"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "919fee17e21b2043b331b01775773c22"
server_key_iv: "649a27cc2c210d735edd3c6aadeef915"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.626127 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "aa6e8b5eeba74eefa4eb7defaeada375"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "919fee17e21b2043b331b01775773c22"
server_key_iv: "649a27cc2c210d735edd3c6aadeef915"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.629518 12485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:09.631655 12499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.632859 12485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:09.633009 12485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "aa6e8b5eeba74eefa4eb7defaeada375"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "919fee17e21b2043b331b01775773c22"
server_key_iv: "649a27cc2c210d735edd3c6aadeef915"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.633119 12485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:09.643427 12485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:09.644068 12485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:09.644250 12485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:09.651041 12485 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:45435
I20260620 14:58:09.651093 12551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:45435 every 8 connection(s)
I20260620 14:58:09.651963 12485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:09.654527 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:09.657527 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 12485
I20260620 14:58:09.657680 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:09.657953 7354 external_mini_cluster.cc:1428] Setting key bbb5c43dc8310a69991b9a3d5f5d1608
I20260620 14:58:09.660205 12552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375: Bootstrap starting.
I20260620 14:58:09.662834 12552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:09.663640 12552 log.cc:824] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:09.665719 12552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375: No bootstrap required, opened a new log
I20260620 14:58:09.668385 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.659256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.7.46.190:45435)
0620 14:58:09.659926 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.659945 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.659985 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:58:09.660849 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.660871 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.661360 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.661731 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.662144 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:09.663614 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:09.664824 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:09.665466 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:09.665634 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:09.666412 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.666418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.666433 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.666920 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.666984 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.667007 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.667110 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.667209 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.667461 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":89,"threads_started":1}
I20260620 14:58:09.670136 12552 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e8b5eeba74eefa4eb7defaeada375" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45435 } }
I20260620 14:58:09.670396 12552 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:09.670459 12552 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e8b5eeba74eefa4eb7defaeada375, State: Initialized, Role: FOLLOWER
I20260620 14:58:09.670900 12552 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e8b5eeba74eefa4eb7defaeada375" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45435 } }
I20260620 14:58:09.671025 12552 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:09.671105 12552 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:09.671224 12552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:09.671307 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:09.672310 12552 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e8b5eeba74eefa4eb7defaeada375" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45435 } }
I20260620 14:58:09.672636 12552 leader_election.cc:302] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa6e8b5eeba74eefa4eb7defaeada375; no voters:
I20260620 14:58:09.672926 12552 leader_election.cc:288] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:09.673053 12557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:09.673280 12557 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [term 1 LEADER]: Becoming Leader. State: Replica: aa6e8b5eeba74eefa4eb7defaeada375, State: Running, Role: LEADER
I20260620 14:58:09.673559 12557 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e8b5eeba74eefa4eb7defaeada375" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45435 } }
I20260620 14:58:09.674105 12552 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:09.675227 12558 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa6e8b5eeba74eefa4eb7defaeada375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e8b5eeba74eefa4eb7defaeada375" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45435 } } }
I20260620 14:58:09.675359 12558 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:09.675601 12559 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa6e8b5eeba74eefa4eb7defaeada375. Latest consensus state: current_term: 1 leader_uuid: "aa6e8b5eeba74eefa4eb7defaeada375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e8b5eeba74eefa4eb7defaeada375" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45435 } } }
I20260620 14:58:09.675698 12559 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:09.675820 12566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:09.678728 12566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:09.684187 12566 catalog_manager.cc:1347] Generated new cluster ID: 3e90140f14a34b249a47ff68dcc82e52
I20260620 14:58:09.684257 12566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:09.699990 12566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:09.701081 12566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:09.704557 12566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa6e8b5eeba74eefa4eb7defaeada375: Generated new TSK 0
I20260620 14:58:09.705247 12566 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:58:09.741925 12582 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:58:09.742031 12582 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:58:09.742440 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.718706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.7.46.190:45435)
0620 14:58:09.718826 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.718832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.719036 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:58:09.719257 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.719260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.719314 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.719419 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:58:09.719670 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:09.720274 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:09.721164 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:09.721392 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:09.721501 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:09.721838 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:58:09.742079 (+ 20241us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:58:09.742328 (+ 249us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35120: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 14:58:09.742537 12555 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35120: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:58:09.743803 12581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:09.718628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45435 (local address 127.0.0.1:35120)
0620 14:58:09.718947 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.718959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:09.719095 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:09.719437 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:09.719440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:09.719448 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:58:09.719562 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:09.719568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:09.720395 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:09.720399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:09.721036 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:09.721043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:09.721508 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:09.721512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:09.721612 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:09.721653 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:58:09.742294 (+ 20641us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:58:09.742523 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.46.190:45435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.46.190:45435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:58:09.744881 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 12485
2026-06-20T14:58:09Z chronyd exiting
I20260620 14:58:09.781302 7354 test_util.cc:175] -----------------------------------------------
I20260620 14:58:09.781381 7354 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967464421185-7354-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:58:09.782424 7354 mini_oidc.cc:152] Starting JWKS server
I20260620 14:58:09.783267 7354 webserver.cc:460] Webserver started at https://127.0.0.1:45559/ using document root <none> and password file <none>
I20260620 14:58:09.783380 7354 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:58:09.783689 7354 webserver.cc:460] Webserver started at http://127.0.0.1:46697/ using document root <none> and password file <none>
2026-06-20T14:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:09Z Disabled control of system clock
I20260620 14:58:09.791718 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43073
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43073
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/testchainca.cert with env {}
W20260620 14:58:09.896847 12588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:09.897131 12588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:09.897213 12588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:09.901235 12588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:09.901314 12588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:09.901340 12588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:09.901381 12588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:09.901414 12588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:58:09.901470 12588 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:58:09.901505 12588 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45559/jwks.json
I20260620 14:58:09.906874 12588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:09.907896 12588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:09.908828 12588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:09.916955 12594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.916946 12596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:09.917006 12588 server_base.cc:1034] running on GCE node
W20260620 14:58:09.916946 12593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:09.917768 12588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:09.918766 12588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:09.919943 12588 hybrid_clock.cc:648] HybridClock initialized: now 1781967489919937 us; error 29 us; skew 500 ppm
I20260620 14:58:09.921605 12588 webserver.cc:460] Webserver started at http://127.7.46.190:45181/ using document root <none> and password file <none>
I20260620 14:58:09.922156 12588 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:09.922243 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:09.922439 12588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:09.924467 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "b0303861a38f48a6b6d9070f1f5cd193"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "51c8e35e2586edb3dff8336e35bc50e2"
server_key_iv: "916bf77b7cd4797a76ab2fdc54ffffbb"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.925009 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "b0303861a38f48a6b6d9070f1f5cd193"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "51c8e35e2586edb3dff8336e35bc50e2"
server_key_iv: "916bf77b7cd4797a76ab2fdc54ffffbb"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.928537 12588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:09.930761 12602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.931972 12588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:58:09.932111 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "b0303861a38f48a6b6d9070f1f5cd193"
format_stamp: "Formatted at 2026-06-20 14:58:09 on dist-test-slave-30t4"
server_key: "51c8e35e2586edb3dff8336e35bc50e2"
server_key_iv: "916bf77b7cd4797a76ab2fdc54ffffbb"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.932219 12588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:09.948904 12588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:09.949523 12588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:09.949710 12588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:09.956099 12588 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:43073
I20260620 14:58:09.956130 12654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:43073 every 8 connection(s)
I20260620 14:58:09.957062 12588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:09.960144 12655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:09.965353 12655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193: Bootstrap starting.
I20260620 14:58:09.966921 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 12588
I20260620 14:58:09.967015 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:09.967272 7354 external_mini_cluster.cc:1428] Setting key 7be2c9740facc799f5d219441f967ac8
I20260620 14:58:09.967859 12655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:09.968734 12655 log.cc:824] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:09.970810 12655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193: No bootstrap required, opened a new log
I20260620 14:58:09.975198 12655 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0303861a38f48a6b6d9070f1f5cd193" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43073 } }
I20260620 14:58:09.975440 12655 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:09.975505 12655 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0303861a38f48a6b6d9070f1f5cd193, State: Initialized, Role: FOLLOWER
I20260620 14:58:09.976006 12655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0303861a38f48a6b6d9070f1f5cd193" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43073 } }
I20260620 14:58:09.976132 12655 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:09.976215 12655 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:09.976334 12655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:09.976981 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.968348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.7.46.190:43073)
0620 14:58:09.968987 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:09.969003 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:09.969038 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:58:09.969815 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:09.969834 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:09.970153 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:09.970420 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:09.970874 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:09.972183 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:09.973437 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:09.973899 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:09.974040 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:09.974972 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:09.974978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:09.974990 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:09.975365 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:09.975417 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:09.975435 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:09.975562 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:09.975717 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:58:09.976149 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20260620 14:58:09.977268 12655 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0303861a38f48a6b6d9070f1f5cd193" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43073 } }
I20260620 14:58:09.977597 12655 leader_election.cc:302] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0303861a38f48a6b6d9070f1f5cd193; no voters:
I20260620 14:58:09.977922 12655 leader_election.cc:288] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:09.978035 12660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:09.978219 12660 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [term 1 LEADER]: Becoming Leader. State: Replica: b0303861a38f48a6b6d9070f1f5cd193, State: Running, Role: LEADER
I20260620 14:58:09.978505 12660 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0303861a38f48a6b6d9070f1f5cd193" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43073 } }
I20260620 14:58:09.979141 12655 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:09.979653 12660 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0303861a38f48a6b6d9070f1f5cd193. Latest consensus state: current_term: 1 leader_uuid: "b0303861a38f48a6b6d9070f1f5cd193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0303861a38f48a6b6d9070f1f5cd193" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43073 } } }
I20260620 14:58:09.979758 12660 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:09.979967 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:09.980041 12662 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0303861a38f48a6b6d9070f1f5cd193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0303861a38f48a6b6d9070f1f5cd193" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43073 } } }
I20260620 14:58:09.980129 12662 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:09.980445 12665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:09.983596 12665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:09.988932 12665 catalog_manager.cc:1347] Generated new cluster ID: f37ff7d75cfe48f0afbdaad8f0d8d4c4
I20260620 14:58:09.988999 12665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:10.024937 12665 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:10.025985 12665 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:10.035971 12665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0303861a38f48a6b6d9070f1f5cd193: Generated new TSK 0
I20260620 14:58:10.036823 12665 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:58:10.064033 12685 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:58:10.064142 12685 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:58:10.064761 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.048497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.7.46.190:43073)
0620 14:58:10.048634 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:10.048638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:10.048780 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:58:10.048899 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:10.048902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:10.048958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:10.049072 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:58:10.049267 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.050103 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.050928 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.051283 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.051389 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:10.051667 (+ 278us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:58:10.064369 (+ 12702us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:58:10.064624 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34928: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:58:10.064831 12658 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34928: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:58:10.064839 12684 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:10.048379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43073 (local address 127.0.0.1:34928)
0620 14:58:10.048695 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.048706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:10.048795 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:10.049074 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:10.049076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:10.049082 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:58:10.049167 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:10.049171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.050202 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.050204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:10.050816 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:10.050823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.051406 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.051408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:10.051482 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:10.051502 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:58:10.064559 (+ 13057us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:58:10.064701 (+ 142us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.46.190:43073: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260620 14:58:10.065512 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 12588
2026-06-20T14:58:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:58:10.104642 7354 mini_oidc.cc:152] Starting JWKS server
I20260620 14:58:10.105574 7354 webserver.cc:460] Webserver started at https://127.0.0.1:36675/ using document root <none> and password file <none>
I20260620 14:58:10.105655 7354 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:58:10.105904 7354 webserver.cc:460] Webserver started at http://127.0.0.1:34909/ using document root <none> and password file <none>
2026-06-20T14:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:10Z Disabled control of system clock
I20260620 14:58:10.114252 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:46493
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:46493
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36675/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:58:10.219250 12691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:10.219532 12691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:10.219610 12691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:10.223474 12691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:10.223560 12691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:10.223587 12691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:10.223626 12691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:10.223659 12691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:58:10.223708 12691 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:58:10.223744 12691 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36675/jwks.json
I20260620 14:58:10.228303 12691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:46493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:46493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36675/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:10.229306 12691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:10.230614 12691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:10.238777 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:10.238799 12699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:10.238773 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:10.239117 12691 server_base.cc:1034] running on GCE node
I20260620 14:58:10.239665 12691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:10.241149 12691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:10.242357 12691 hybrid_clock.cc:648] HybridClock initialized: now 1781967490242326 us; error 41 us; skew 500 ppm
I20260620 14:58:10.244501 12691 webserver.cc:460] Webserver started at http://127.7.46.190:45383/ using document root <none> and password file <none>
I20260620 14:58:10.245082 12691 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:10.245149 12691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:10.245378 12691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:10.247396 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "c7651133a1b640d0a313ac522bb7595b"
format_stamp: "Formatted at 2026-06-20 14:58:10 on dist-test-slave-30t4"
server_key: "c4e4e77ada09f16671ca930d738c6930"
server_key_iv: "efe3c649c4c0936442738a6da0910d68"
server_key_version: "encryptionkey@0"
I20260620 14:58:10.247991 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "c7651133a1b640d0a313ac522bb7595b"
format_stamp: "Formatted at 2026-06-20 14:58:10 on dist-test-slave-30t4"
server_key: "c4e4e77ada09f16671ca930d738c6930"
server_key_iv: "efe3c649c4c0936442738a6da0910d68"
server_key_version: "encryptionkey@0"
I20260620 14:58:10.251682 12691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:58:10.254078 12705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:10.255322 12691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:10.255463 12691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "c7651133a1b640d0a313ac522bb7595b"
format_stamp: "Formatted at 2026-06-20 14:58:10 on dist-test-slave-30t4"
server_key: "c4e4e77ada09f16671ca930d738c6930"
server_key_iv: "efe3c649c4c0936442738a6da0910d68"
server_key_version: "encryptionkey@0"
I20260620 14:58:10.255564 12691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:10.277289 12691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:10.277933 12691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:10.278124 12691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:10.284551 12691 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:46493
I20260620 14:58:10.284590 12757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:46493 every 8 connection(s)
I20260620 14:58:10.285447 12691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:10.288709 12758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:10.289601 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 12691
I20260620 14:58:10.289733 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:10.290010 7354 external_mini_cluster.cc:1428] Setting key eececd50f023db4c5be0b92759a6431a
I20260620 14:58:10.295140 12758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b: Bootstrap starting.
I20260620 14:58:10.297600 12758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:10.298420 12758 log.cc:824] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:10.299131 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.291503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.7.46.190:46493)
0620 14:58:10.292179 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:10.292198 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:10.292233 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:58:10.292872 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:10.292888 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:10.293276 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:10.293542 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:10.293853 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.294787 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.295958 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.296398 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.296509 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:10.297290 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:10.297295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:10.297306 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:10.297662 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:10.297753 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:10.297787 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:10.297861 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:10.298019 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:58:10.298300 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":116,"threads_started":1}
I20260620 14:58:10.300462 12758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b: No bootstrap required, opened a new log
I20260620 14:58:10.301789 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:10.304198 12758 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7651133a1b640d0a313ac522bb7595b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46493 } }
I20260620 14:58:10.304423 12758 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:10.304500 12758 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7651133a1b640d0a313ac522bb7595b, State: Initialized, Role: FOLLOWER
I20260620 14:58:10.304884 12758 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7651133a1b640d0a313ac522bb7595b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46493 } }
I20260620 14:58:10.305032 12758 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:10.305114 12758 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:10.305228 12758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:10.306046 12758 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7651133a1b640d0a313ac522bb7595b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46493 } }
I20260620 14:58:10.306382 12758 leader_election.cc:302] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7651133a1b640d0a313ac522bb7595b; no voters:
I20260620 14:58:10.306702 12758 leader_election.cc:288] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:10.306828 12763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:10.307042 12763 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [term 1 LEADER]: Becoming Leader. State: Replica: c7651133a1b640d0a313ac522bb7595b, State: Running, Role: LEADER
I20260620 14:58:10.307333 12763 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7651133a1b640d0a313ac522bb7595b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46493 } }
I20260620 14:58:10.307906 12758 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:10.308907 12764 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7651133a1b640d0a313ac522bb7595b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7651133a1b640d0a313ac522bb7595b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46493 } } }
I20260620 14:58:10.309043 12764 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:10.309330 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7651133a1b640d0a313ac522bb7595b. Latest consensus state: current_term: 1 leader_uuid: "c7651133a1b640d0a313ac522bb7595b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7651133a1b640d0a313ac522bb7595b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 46493 } } }
I20260620 14:58:10.309422 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:10.309788 12772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:10.312327 12772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:10.317492 12772 catalog_manager.cc:1347] Generated new cluster ID: 6fb8f9c9a8a5448a83ce3d4478ac0499
I20260620 14:58:10.317565 12772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:10.338438 12772 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:10.339349 12772 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:10.343552 12772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7651133a1b640d0a313ac522bb7595b: Generated new TSK 0
I20260620 14:58:10.344214 12772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:10.357234 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.352561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.7.46.190:46493)
0620 14:58:10.352694 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:10.352700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:10.352897 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:58:10.353080 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:10.353083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:10.353129 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:10.353233 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:58:10.353480 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.354062 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.354934 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.355161 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.355267 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:10.355576 (+ 309us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:58:10.356886 (+ 1310us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36675/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:58:10.357106 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51172: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36675/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 14:58:10.357290 12761 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51172: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36675/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:58:10.359086 12787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:10.352481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:46493 (local address 127.0.0.1:51172)
0620 14:58:10.352809 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.352820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:10.352939 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:10.353243 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:10.353246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:10.353254 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:58:10.353371 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:10.353377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.354169 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.354174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:10.354811 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:10.354819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.355287 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.355291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:10.355384 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:10.355404 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:58:10.357050 (+ 1646us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36675/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:58:10.358920 (+ 1870us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.46.190:46493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36675/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1663,"thread_start_us":102,"threads_started":1}
I20260620 14:58:10.359652 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 12691
E20260620 14:58:10.368161 12788 webserver.cc:533] Webserver: sslize failed:
E20260620 14:58:10.368242 12788 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:58:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:10Z Disabled control of system clock
I20260620 14:58:10.406379 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44515
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:44515
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/testchainca.cert with env {}
W20260620 14:58:10.513199 12792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:10.513453 12792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:10.513490 12792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:10.517376 12792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:10.517426 12792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:10.517441 12792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:10.517457 12792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:10.517472 12792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:58:10.517488 12792 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:58:10.517503 12792 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:58:10.522673 12792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:44515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:10.523617 12792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:10.524612 12792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:10.532621 12800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:10.532620 12798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:10.532692 12797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:10.533087 12792 server_base.cc:1034] running on GCE node
I20260620 14:58:10.533605 12792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:10.534606 12792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:10.535832 12792 hybrid_clock.cc:648] HybridClock initialized: now 1781967490535791 us; error 46 us; skew 500 ppm
I20260620 14:58:10.537652 12792 webserver.cc:460] Webserver started at http://127.7.46.190:39601/ using document root <none> and password file <none>
I20260620 14:58:10.538220 12792 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:10.538312 12792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:10.538532 12792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:10.540549 12792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "eed409eed5ed411ebff3f7403cc7060e"
format_stamp: "Formatted at 2026-06-20 14:58:10 on dist-test-slave-30t4"
server_key: "6f55f3651f7cc53e647fa3d7b8c09986"
server_key_iv: "4403aace5fc060fce8ed9c6033bbd5a1"
server_key_version: "encryptionkey@0"
I20260620 14:58:10.541107 12792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "eed409eed5ed411ebff3f7403cc7060e"
format_stamp: "Formatted at 2026-06-20 14:58:10 on dist-test-slave-30t4"
server_key: "6f55f3651f7cc53e647fa3d7b8c09986"
server_key_iv: "4403aace5fc060fce8ed9c6033bbd5a1"
server_key_version: "encryptionkey@0"
I20260620 14:58:10.544662 12792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:10.546943 12806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:10.548178 12792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:10.548308 12792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "eed409eed5ed411ebff3f7403cc7060e"
format_stamp: "Formatted at 2026-06-20 14:58:10 on dist-test-slave-30t4"
server_key: "6f55f3651f7cc53e647fa3d7b8c09986"
server_key_iv: "4403aace5fc060fce8ed9c6033bbd5a1"
server_key_version: "encryptionkey@0"
I20260620 14:58:10.548413 12792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:10.569609 12792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:10.570326 12792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:10.570521 12792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:10.577468 12792 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:44515
I20260620 14:58:10.577477 12858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:44515 every 8 connection(s)
I20260620 14:58:10.578377 12792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:10.581362 12859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:10.582090 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 12792
I20260620 14:58:10.582228 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:10.582515 7354 external_mini_cluster.cc:1428] Setting key 457fd94f3556ef144e5589fd92eab3ac
I20260620 14:58:10.587710 12859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e: Bootstrap starting.
I20260620 14:58:10.590142 12859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:10.591001 12859 log.cc:824] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:10.592348 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.583891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.7.46.190:44515)
0620 14:58:10.584547 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:10.584566 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:10.584607 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:58:10.585411 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:10.585433 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:10.585908 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:10.586267 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:10.586705 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.588107 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.588920 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.589569 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.589719 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:10.590475 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:10.590481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:10.590496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:10.590979 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:10.591043 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:10.591071 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:10.591163 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:10.591328 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:58:10.591586 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260620 14:58:10.593005 12859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e: No bootstrap required, opened a new log
I20260620 14:58:10.594907 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:10.597007 12859 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed409eed5ed411ebff3f7403cc7060e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44515 } }
I20260620 14:58:10.597244 12859 raft_consensus.cc:374] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:10.597304 12859 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed409eed5ed411ebff3f7403cc7060e, State: Initialized, Role: FOLLOWER
I20260620 14:58:10.597694 12859 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed409eed5ed411ebff3f7403cc7060e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44515 } }
I20260620 14:58:10.597859 12859 raft_consensus.cc:388] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:10.597926 12859 raft_consensus.cc:482] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:10.598042 12859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:10.598932 12859 raft_consensus.cc:504] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed409eed5ed411ebff3f7403cc7060e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44515 } }
I20260620 14:58:10.599303 12859 leader_election.cc:302] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eed409eed5ed411ebff3f7403cc7060e; no voters:
I20260620 14:58:10.599650 12859 leader_election.cc:288] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:10.599725 12864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:10.600023 12864 raft_consensus.cc:686] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [term 1 LEADER]: Becoming Leader. State: Replica: eed409eed5ed411ebff3f7403cc7060e, State: Running, Role: LEADER
I20260620 14:58:10.600283 12864 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed409eed5ed411ebff3f7403cc7060e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44515 } }
I20260620 14:58:10.600841 12859 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:10.601955 12865 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eed409eed5ed411ebff3f7403cc7060e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed409eed5ed411ebff3f7403cc7060e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44515 } } }
I20260620 14:58:10.602092 12865 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:10.602306 12866 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed409eed5ed411ebff3f7403cc7060e. Latest consensus state: current_term: 1 leader_uuid: "eed409eed5ed411ebff3f7403cc7060e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed409eed5ed411ebff3f7403cc7060e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44515 } } }
I20260620 14:58:10.602381 12866 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:10.602452 12873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:10.605224 12873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:10.610132 12873 catalog_manager.cc:1347] Generated new cluster ID: a0d0721a23ef4657983e8162172ed200
I20260620 14:58:10.610191 12873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:10.618307 12873 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:10.619367 12873 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:10.625139 12873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eed409eed5ed411ebff3f7403cc7060e: Generated new TSK 0
I20260620 14:58:10.625804 12873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:10.710939 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.636735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.7.46.190:44515)
0620 14:58:10.636858 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:10.636863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:10.637001 (+ 138us) server_negotiation.cc:407] Connection header received
0620 14:58:10.637118 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:10.637121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:10.637172 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:10.637282 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:58:10.637473 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.638281 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.639111 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.639453 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.639547 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:10.639821 (+ 274us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:58:10.710449 (+ 70628us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:58:10.710778 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59458: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 14:58:10.711027 12862 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59458: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:58:10.711097 12888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:10.636632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44515 (local address 127.0.0.1:59458)
0620 14:58:10.636922 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.636934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:10.637021 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:10.637284 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:10.637286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:10.637292 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:58:10.637377 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:10.637381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.638403 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.638405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:10.639007 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:10.639013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:10.639565 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:10.639568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:10.639625 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:10.639644 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:58:10.710687 (+ 71043us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:58:10.710877 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.46.190:44515: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
I20260620 14:58:10.711849 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 12792
2026-06-20T14:58:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (332 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:11 dist-test-slave-30t4 krb5kdc[12900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:11 dist-test-slave-30t4 krb5kdc[12900](info): set up 2 sockets
Jun 20 14:58:11 dist-test-slave-30t4 krb5kdc[12900](info): commencing operation
krb5kdc: starting...
W20260620 14:58:11.988093 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:58:12.102710 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37661
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:37661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:12.208336 12913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:12.208603 12913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:12.208675 12913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:12.212797 12913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:12.212879 12913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:12.212917 12913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:12.212954 12913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:12.212990 12913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:12.218094 12913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:37661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:12.219100 12913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:12.220116 12913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:12.228669 12919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:12.228735 12913 server_base.cc:1034] running on GCE node
W20260620 14:58:12.228664 12921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:12.228663 12918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:12.229463 12913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:12.230468 12913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:12.231664 12913 hybrid_clock.cc:648] HybridClock initialized: now 1781967492231655 us; error 28 us; skew 500 ppm
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.234380 12913 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:58:12.235195 12913 webserver.cc:460] Webserver started at http://127.7.46.190:44689/ using document root <none> and password file <none>
I20260620 14:58:12.235723 12913 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:12.235838 12913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:12.236034 12913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:12.237967 12913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "ea4aa8f7a5e44268b6e353822788bb57"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "9147a50e919b80c37918f0a0c1c184b2"
server_key_iv: "0fac7d5c94b768c4b9bac797b60f45a8"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.238457 12913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "ea4aa8f7a5e44268b6e353822788bb57"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "9147a50e919b80c37918f0a0c1c184b2"
server_key_iv: "0fac7d5c94b768c4b9bac797b60f45a8"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.241925 12913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:58:12.244148 12928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.245298 12913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:12.245391 12913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "ea4aa8f7a5e44268b6e353822788bb57"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "9147a50e919b80c37918f0a0c1c184b2"
server_key_iv: "0fac7d5c94b768c4b9bac797b60f45a8"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.245468 12913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:12.273496 12913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:12.276300 12913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:12.276451 12913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:12.283129 12913 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:37661
I20260620 14:58:12.283154 12980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:37661 every 8 connection(s)
I20260620 14:58:12.284075 12913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:12.286634 12981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:12.288486 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 12913
I20260620 14:58:12.288614 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:12.288890 7354 external_mini_cluster.cc:1428] Setting key bb6d8f24bbb1aae95332da8aebebae98
I20260620 14:58:12.292727 12981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57: Bootstrap starting.
I20260620 14:58:12.294798 12981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:12.295482 12981 log.cc:824] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:12.297204 12981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57: No bootstrap required, opened a new log
I20260620 14:58:12.300374 12981 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4aa8f7a5e44268b6e353822788bb57" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37661 } }
I20260620 14:58:12.300601 12981 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:12.300678 12981 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4aa8f7a5e44268b6e353822788bb57, State: Initialized, Role: FOLLOWER
I20260620 14:58:12.301156 12981 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4aa8f7a5e44268b6e353822788bb57" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37661 } }
I20260620 14:58:12.301313 12981 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:12.301388 12981 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:12.301509 12981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:12.302186 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.290190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.7.46.190:37661)
0620 14:58:12.290592 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:12.290606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:12.290637 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:58:12.291238 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:12.291253 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:12.291547 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:12.291885 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:12.292668 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.293636 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.294460 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.294965 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.295142 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:12.297623 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:12.297628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:12.297639 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:12.297664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:12.299312 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.299758 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.299794 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.299802 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.299912 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.300205 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.300208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.300209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.300568 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:12.300787 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:12.301098 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:58:12.301396 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260620 14:58:12.302342 12981 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4aa8f7a5e44268b6e353822788bb57" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37661 } }
I20260620 14:58:12.302654 12981 leader_election.cc:302] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea4aa8f7a5e44268b6e353822788bb57; no voters:
I20260620 14:58:12.302987 12981 leader_election.cc:288] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:12.303809 12986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:12.304077 12986 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [term 1 LEADER]: Becoming Leader. State: Replica: ea4aa8f7a5e44268b6e353822788bb57, State: Running, Role: LEADER
I20260620 14:58:12.304169 12981 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:12.304363 12986 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4aa8f7a5e44268b6e353822788bb57" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37661 } }
I20260620 14:58:12.305876 12988 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4aa8f7a5e44268b6e353822788bb57. Latest consensus state: current_term: 1 leader_uuid: "ea4aa8f7a5e44268b6e353822788bb57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4aa8f7a5e44268b6e353822788bb57" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37661 } } }
I20260620 14:58:12.305997 12988 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:12.306281 12987 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea4aa8f7a5e44268b6e353822788bb57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4aa8f7a5e44268b6e353822788bb57" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37661 } } }
I20260620 14:58:12.306581 12987 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57 [sys.catalog]: This master's current role is: LEADER
W20260620 14:58:12.308691 13000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:58:12.308779 13000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:58:12.308853 13001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:12.309633 13001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:12.314603 13001 catalog_manager.cc:1347] Generated new cluster ID: 889053a7579448b7b7abd2cc532cb5b9
I20260620 14:58:12.314683 13001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:12.320917 13001 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:12.321847 13001 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:12.328102 13001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea4aa8f7a5e44268b6e353822788bb57: Generated new TSK 0
I20260620 14:58:12.328707 13001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:58:12.354038 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37661
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:12.462363 13007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:12.462678 13007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:12.462752 13007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:12.466809 13007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:12.466902 13007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:12.467000 13007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:12.472313 13007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:12.473369 13007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:12.474377 13007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:12.483809 13013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:12.483850 13012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:12.483861 13015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:12.483942 13007 server_base.cc:1034] running on GCE node
I20260620 14:58:12.484455 13007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:12.485077 13007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:12.486238 13007 hybrid_clock.cc:648] HybridClock initialized: now 1781967492486225 us; error 24 us; skew 500 ppm
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.488986 13007 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:58:12.489732 13007 webserver.cc:460] Webserver started at http://127.7.46.129:42169/ using document root <none> and password file <none>
I20260620 14:58:12.490226 13007 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:12.490278 13007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:12.490433 13007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:12.492424 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "9ddffc9ba2f745b4b9b1519460dca586"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "7b3fa0d5f2e9fb1440be0f6ce249962e"
server_key_iv: "e4ecac32a03c7803f20a8bf47ba87dd5"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.492930 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "9ddffc9ba2f745b4b9b1519460dca586"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "7b3fa0d5f2e9fb1440be0f6ce249962e"
server_key_iv: "e4ecac32a03c7803f20a8bf47ba87dd5"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.496443 13007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:12.498719 13022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.499954 13007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:12.500077 13007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "9ddffc9ba2f745b4b9b1519460dca586"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "7b3fa0d5f2e9fb1440be0f6ce249962e"
server_key_iv: "e4ecac32a03c7803f20a8bf47ba87dd5"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.500180 13007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:12.514269 13007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:12.517450 13007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:12.517647 13007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:12.518215 13007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:12.518998 13007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:12.519068 13007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.519132 13007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:12.519181 13007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.527456 13007 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:45545
I20260620 14:58:12.527514 13135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:45545 every 8 connection(s)
I20260620 14:58:12.528393 13007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:12.530176 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 13007
I20260620 14:58:12.530323 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:12.530640 7354 external_mini_cluster.cc:1428] Setting key 51158affd8c3d13e6a942546c863bc04
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 14:58:12.542928 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.530106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:47999 (local address 127.7.46.190:37661)
0620 14:58:12.530227 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:12.530231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:12.531162 (+ 931us) server_negotiation.cc:407] Connection header received
0620 14:58:12.532616 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:12.532620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:12.532670 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:12.532745 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:12.533836 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.534389 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.536183 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.536415 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.536514 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:12.539307 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:12.539311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:12.539319 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:12.539341 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:12.540946 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.541440 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.541442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.541444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.541516 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.541810 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.541812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.541813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.541994 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:12.542071 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:12.542630 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 14:58:12.542756 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:58:12.543988 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.530579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37661 (local address 127.7.46.129:47999)
0620 14:58:12.531022 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:12.531071 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:12.532413 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:12.532909 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:12.532916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:12.533246 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:12.533690 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:12.533702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.534518 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.534524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:12.536063 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:12.536076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.536583 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.536587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:12.536817 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:12.537446 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:12.537471 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:12.539139 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:12.541048 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:12.541053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:12.541064 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:12.541314 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:12.541598 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:12.541600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:12.541602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:12.541728 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:12.542086 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:12.542091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:12.542428 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 14:58:12.542587 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 14:58:12.542950 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":128,"threads_started":1}
I20260620 14:58:12.545269 13136 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37661
I20260620 14:58:12.545512 13136 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:12.545989 13136 heartbeater.cc:511] Master 127.7.46.190:37661 requested a full tablet report, sending...
I20260620 14:58:12.547389 12945 ts_manager.cc:194] Registered new tserver with Master: 9ddffc9ba2f745b4b9b1519460dca586 (127.7.46.129:45545)
I20260620 14:58:12.548539 12945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:47999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:58:12.558164 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37661
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:12.664117 13141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:12.664367 13141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:12.664410 13141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:12.668248 13141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:12.668292 13141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:12.668345 13141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:12.673327 13141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:12.674247 13141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:12.675155 13141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:12.684450 13146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:12.684450 13147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:12.684468 13149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:12.685118 13141 server_base.cc:1034] running on GCE node
I20260620 14:58:12.685477 13141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:12.686058 13141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:12.687240 13141 hybrid_clock.cc:648] HybridClock initialized: now 1781967492687208 us; error 47 us; skew 500 ppm
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.689779 13141 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:58:12.690624 13141 webserver.cc:460] Webserver started at http://127.7.46.130:36221/ using document root <none> and password file <none>
I20260620 14:58:12.691149 13141 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:12.691237 13141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:12.691452 13141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:12.693449 13141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "75987790b8e84a5dbb7af77bcf6b629e"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "0a54b90ba3d48a63376a93498212a492"
server_key_iv: "13c86fcaa4cda32715f2d43cabe271b5"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.693990 13141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "75987790b8e84a5dbb7af77bcf6b629e"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "0a54b90ba3d48a63376a93498212a492"
server_key_iv: "13c86fcaa4cda32715f2d43cabe271b5"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.697507 13141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:12.699651 13156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.700809 13141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:12.700946 13141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "75987790b8e84a5dbb7af77bcf6b629e"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "0a54b90ba3d48a63376a93498212a492"
server_key_iv: "13c86fcaa4cda32715f2d43cabe271b5"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.701071 13141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:12.718775 13141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:12.721933 13141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:12.722131 13141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:12.722687 13141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:12.723508 13141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:12.723569 13141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.723635 13141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:12.723672 13141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.732165 13141 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:38647
I20260620 14:58:12.732187 13269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:38647 every 8 connection(s)
I20260620 14:58:12.733011 13141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:12.733845 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 13141
I20260620 14:58:12.734000 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:12.734308 7354 external_mini_cluster.cc:1428] Setting key 207e932189fea0491d40b963a8388eb8
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
I20260620 14:58:12.747056 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.734947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:38391 (local address 127.7.46.190:37661)
0620 14:58:12.735086 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:12.735090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:12.736043 (+ 953us) server_negotiation.cc:407] Connection header received
0620 14:58:12.737374 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:12.737377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:12.737425 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:12.737518 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:12.738604 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.739125 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.740300 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.740535 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.740647 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:12.742946 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:12.742949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:12.742951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:12.742973 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:12.744488 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.745020 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.745023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.745025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.745104 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.745519 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.745522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.745523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.745696 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:12.745810 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:12.746495 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 14:58:12.746712 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:58:12.748001 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.735343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37661 (local address 127.7.46.130:38391)
0620 14:58:12.735893 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:12.735935 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:12.737171 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:12.737676 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:12.737683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:12.738020 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:12.738461 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:12.738473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.739248 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.739252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:12.740178 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:12.740188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.740680 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.740684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:12.740840 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:12.741430 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:12.741445 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:12.742776 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:12.744626 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:12.744632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:12.744646 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:12.744912 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:12.745227 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:12.745230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:12.745232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:12.745362 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:12.745808 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:12.745815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:12.746156 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 14:58:12.746380 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 14:58:12.746737 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":133,"threads_started":1}
I20260620 14:58:12.749346 13270 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37661
I20260620 14:58:12.749589 13270 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:12.750164 13270 heartbeater.cc:511] Master 127.7.46.190:37661 requested a full tablet report, sending...
I20260620 14:58:12.751376 12945 ts_manager.cc:194] Registered new tserver with Master: 75987790b8e84a5dbb7af77bcf6b629e (127.7.46.130:38647)
I20260620 14:58:12.752240 12945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:38391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:58:12.764925 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37661
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:12.873126 13275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:12.873401 13275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:12.873481 13275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:12.877535 13275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:12.877617 13275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:12.877707 13275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:12.883287 13275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:12.884387 13275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:12.885362 13275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:12.894735 13280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:12.894786 13275 server_base.cc:1034] running on GCE node
W20260620 14:58:12.894770 13283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:12.894735 13281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:12.895361 13275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:12.896035 13275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:12.897223 13275 hybrid_clock.cc:648] HybridClock initialized: now 1781967492897210 us; error 29 us; skew 500 ppm
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.900007 13275 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:58:12.900825 13275 webserver.cc:460] Webserver started at http://127.7.46.131:46607/ using document root <none> and password file <none>
I20260620 14:58:12.901363 13275 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:12.901453 13275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:12.901669 13275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:12.903723 13275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "938cd2cca9dc4f98b254860de1716887"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "4a46405c4c9e38ffe62be344cde74fa8"
server_key_iv: "76fd6cf7b244f0404559f0461eb73d90"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.904448 13275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "938cd2cca9dc4f98b254860de1716887"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "4a46405c4c9e38ffe62be344cde74fa8"
server_key_iv: "76fd6cf7b244f0404559f0461eb73d90"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.908279 13275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:58:12.910621 13290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.911942 13275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:12.912077 13275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "938cd2cca9dc4f98b254860de1716887"
format_stamp: "Formatted at 2026-06-20 14:58:12 on dist-test-slave-30t4"
server_key: "4a46405c4c9e38ffe62be344cde74fa8"
server_key_iv: "76fd6cf7b244f0404559f0461eb73d90"
server_key_version: "encryptionkey@0"
I20260620 14:58:12.912177 13275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:12.922228 13275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:12.925009 13275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:12.925200 13275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:12.925758 13275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:12.926539 13275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:12.926584 13275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.926653 13275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:12.926684 13275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:12.935395 13275 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:41049
I20260620 14:58:12.935462 13403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:41049 every 8 connection(s)
I20260620 14:58:12.936365 13275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:12.940738 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 13275
I20260620 14:58:12.940852 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:12.941165 7354 external_mini_cluster.cc:1428] Setting key 606c6a7666b412d5cc01c96ee7cd6582
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:12.950003 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.938141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:59937 (local address 127.7.46.190:37661)
0620 14:58:12.938282 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:12.938285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:12.939076 (+ 791us) server_negotiation.cc:407] Connection header received
0620 14:58:12.940447 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:12.940450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:12.940496 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:12.940573 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:12.941999 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.942641 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.944008 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.944217 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.944299 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:12.946774 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:12.946777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:12.946779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:12.946802 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:12.948085 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.948553 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.948555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.948556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.948625 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.948970 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.948973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.948974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.949142 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:12.949219 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:12.949732 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 14:58:12.949830 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:12.950773 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.938510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37661 (local address 127.7.46.131:59937)
0620 14:58:12.938926 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:12.938967 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:12.940260 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:12.940760 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:12.940770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:12.941263 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:12.941840 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:12.941859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.942762 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.942766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:12.943902 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:12.943912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.944340 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.944343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:12.944485 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:12.945077 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:12.945093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:12.946626 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:12.948191 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:12.948196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:12.948207 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:12.948463 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:12.948723 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:12.948726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:12.948728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:12.948883 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:12.949229 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:12.949234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:12.949525 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 14:58:12.949684 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 14:58:12.949945 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260620 14:58:12.951897 13404 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37661
I20260620 14:58:12.952100 13404 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:12.952550 13404 heartbeater.cc:511] Master 127.7.46.190:37661 requested a full tablet report, sending...
I20260620 14:58:12.953572 12945 ts_manager.cc:194] Registered new tserver with Master: 938cd2cca9dc4f98b254860de1716887 (127.7.46.131:41049)
I20260620 14:58:12.954329 12945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:59937
I20260620 14:58:12.955077 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/test-admin.keytab.
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.972941 7354 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:58:12.975903 13411 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:12.974382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.7.46.129:45545)
0620 14:58:12.974717 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:12.974732 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:12.974768 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:58:12.974822 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:12.974827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:12.975026 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:12.975158 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:12.975697 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46926: BlockingRecv error: recv got EOF from 127.0.0.1:46926 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":143,"threads_started":1}
W20260620 14:58:12.976159 7354 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5cc)
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.977337 7354 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.978405 7354 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:58:12.988039 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.979378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.7.46.129:45545)
0620 14:58:12.979555 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:12.979558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:12.979600 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:58:12.979719 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:12.979721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:12.979758 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:12.979873 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:12.980395 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.981173 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.982028 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.982373 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.982453 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:12.984844 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:12.984852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:12.984864 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:12.984894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:12.986394 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.986778 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.986784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.986785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.986859 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:12.987096 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:12.987098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:12.987100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:12.987326 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:12.987472 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:12.987674 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:58:12.987875 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260620 14:58:12.992038 13411 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:12.990967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.7.46.129:45545)
0620 14:58:12.991102 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:12.991107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:12.991225 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:58:12.991355 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:12.991358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:12.991399 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:12.991482 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:12.991942 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46932: BlockingRecv error: recv got EOF from 127.0.0.1:46932 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 14:58:12.992244 7354 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.993319 7354 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:12.994307 7354 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:58:12 dist-test-slave-30t4 krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:58:13.003281 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.995245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.7.46.129:45545)
0620 14:58:12.995400 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:12.995404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:12.995534 (+ 130us) server_negotiation.cc:407] Connection header received
0620 14:58:12.995673 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:12.995676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:12.995719 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:12.995889 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:12.996394 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.996914 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.997705 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:12.997930 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:12.998038 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:13.000233 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:13.000236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:13.000239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:13.000260 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:13.001631 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:13.002089 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:13.002093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:13.002095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:13.002174 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:13.002445 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:13.002450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:13.002452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:13.002617 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:13.002761 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:13.003001 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:58:13.003109 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 14:58:13.006088 13411 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:13.005252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.7.46.129:45545)
0620 14:58:13.005354 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:13.005358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:13.005508 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:58:13.005579 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:13.005581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:13.005614 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:13.005670 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:13.005991 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46946: BlockingRecv error: recv got EOF from 127.0.0.1:46946 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:58:13.006181 7354 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967464421185-7354-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:58:13 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:13.007193 7354 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:58:13 dist-test-slave-30t4 krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:13.008567 7354 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:58:13 dist-test-slave-30t4 krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 14:58:13.017127 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:13.009363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.7.46.129:45545)
0620 14:58:13.009484 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:13.009489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:13.009587 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:58:13.009722 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:13.009725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:13.009762 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:13.009855 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:13.010324 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:13.010809 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:13.011601 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:13.011921 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:13.012034 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:13.014187 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:13.014190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:13.014192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:13.014212 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:13.015748 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:13.016207 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:13.016209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:13.016211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:13.016279 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:13.016500 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:13.016502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:13.016503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:13.016643 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:13.016737 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:13.016917 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:58:13.017014 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:13.018045 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 13007
I20260620 14:58:13.023676 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 13141
I20260620 14:58:13.028846 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 13275
I20260620 14:58:13.033918 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 12913
2026-06-20T14:58:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2087 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:13 dist-test-slave-30t4 krb5kdc[13424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:13 dist-test-slave-30t4 krb5kdc[13424](info): set up 2 sockets
Jun 20 14:58:13 dist-test-slave-30t4 krb5kdc[13424](info): commencing operation
krb5kdc: starting...
W20260620 14:58:14.076227 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:14 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.190.keytab.
Entry for principal oryx/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.190.keytab.
I20260620 14:58:14.194950 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:36619
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.190.keytab
--principal=oryx/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:36619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:14.303675 13437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:14.304004 13437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:14.304054 13437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:14.307873 13437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:14.307916 13437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:14.307935 13437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:14.307951 13437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:14.307966 13437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:14.313265 13437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:36619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.190.keytab
--principal=oryx/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:36619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:14.314173 13437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:14.315182 13437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:14.323589 13443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:14.323598 13445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:14.323625 13442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:14.323800 13437 server_base.cc:1034] running on GCE node
I20260620 14:58:14.324450 13437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:14.325488 13437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:14.326675 13437 hybrid_clock.cc:648] HybridClock initialized: now 1781967494326669 us; error 33 us; skew 500 ppm
Jun 20 14:58:14 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:14.329313 13437 init.cc:377] Logged in from keytab as oryx/127.7.46.190@KRBTEST.COM (short username oryx)
I20260620 14:58:14.330175 13437 webserver.cc:460] Webserver started at http://127.7.46.190:33169/ using document root <none> and password file <none>
I20260620 14:58:14.330704 13437 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:14.330794 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:14.331023 13437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:14.333007 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "6496092390a24fef98320c7d834845bc"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "c50fb1593027b80cbc7297d14cb15616"
server_key_iv: "3a4c0b0b6d586002e84b3a6ed0c24a81"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.333562 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "6496092390a24fef98320c7d834845bc"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "c50fb1593027b80cbc7297d14cb15616"
server_key_iv: "3a4c0b0b6d586002e84b3a6ed0c24a81"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.337152 13437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:14.339484 13452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:14.340746 13437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:14.340886 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "6496092390a24fef98320c7d834845bc"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "c50fb1593027b80cbc7297d14cb15616"
server_key_iv: "3a4c0b0b6d586002e84b3a6ed0c24a81"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.341015 13437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:14.361027 13437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:14.363757 13437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:14.363996 13437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:14.370651 13437 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:36619
I20260620 14:58:14.370942 13504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:36619 every 8 connection(s)
I20260620 14:58:14.371615 13437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:14.374364 13505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:14.379629 13505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: Bootstrap starting.
I20260620 14:58:14.380688 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 13437
I20260620 14:58:14.380846 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:14.381129 7354 external_mini_cluster.cc:1428] Setting key ef259b731a0d92269658bdfb669b7c3c
I20260620 14:58:14.382351 13505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:14.383060 13505 log.cc:824] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:14.385193 13505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: No bootstrap required, opened a new log
I20260620 14:58:14.389508 13505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } }
I20260620 14:58:14.389746 13505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:14.389823 13505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6496092390a24fef98320c7d834845bc, State: Initialized, Role: FOLLOWER
Jun 20 14:58:14 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:14.390334 13505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } }
I20260620 14:58:14.390501 13505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:14.390583 13505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:14.390682 13505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:14.391521 13505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } }
I20260620 14:58:14.391912 13505 leader_election.cc:302] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6496092390a24fef98320c7d834845bc; no voters:
I20260620 14:58:14.392231 13505 leader_election.cc:288] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:14.392333 13510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:14.392546 13510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 1 LEADER]: Becoming Leader. State: Replica: 6496092390a24fef98320c7d834845bc, State: Running, Role: LEADER
I20260620 14:58:14.392866 13510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } }
I20260620 14:58:14.393545 13505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:14.394932 13511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } }
I20260620 14:58:14.395061 13511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:14.395439 13512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6496092390a24fef98320c7d834845bc. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } }
I20260620 14:58:14.395571 13512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:14.395900 13521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:14.396313 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:14.382215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.7.46.190:36619)
0620 14:58:14.382641 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:14.382663 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:14.382697 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:58:14.383368 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:14.383386 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:14.383698 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:14.384041 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:14.384881 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.385890 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.387122 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.387560 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.387670 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:14.390609 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:14.390615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:14.390632 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:14.390662 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:14.392719 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:14.393177 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:14.393182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:14.393188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:14.393334 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:14.393706 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:14.393709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:14.393711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:14.394178 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:14.394377 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:14.395035 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 14:58:14.395340 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":148,"threads_started":1}
I20260620 14:58:14.398940 13521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:14.404866 13521 catalog_manager.cc:1347] Generated new cluster ID: 7c740eac043144dc94b89e76b94c04f0
I20260620 14:58:14.404963 13521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:14.436663 13521 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:14.437773 13521 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:14.444279 13521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: Generated new TSK 0
I20260620 14:58:14.444957 13521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab.
Entry for principal oryx/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab.
I20260620 14:58:14.475121 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36619
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab
--principal=oryx/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:14.596122 13531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:14.596374 13531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:14.596415 13531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:14.600309 13531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:14.600368 13531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:14.600422 13531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:14.605398 13531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab
--principal=oryx/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:14.606290 13531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:14.607198 13531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:14.616297 13537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:14.616293 13536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:14.616302 13539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:14.617206 13531 server_base.cc:1034] running on GCE node
I20260620 14:58:14.617568 13531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:14.618124 13531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:14.619282 13531 hybrid_clock.cc:648] HybridClock initialized: now 1781967494619260 us; error 32 us; skew 500 ppm
Jun 20 14:58:14 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:14.621906 13531 init.cc:377] Logged in from keytab as oryx/127.7.46.129@KRBTEST.COM (short username oryx)
I20260620 14:58:14.622746 13531 webserver.cc:460] Webserver started at http://127.7.46.129:38931/ using document root <none> and password file <none>
I20260620 14:58:14.623286 13531 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:14.623343 13531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:14.623574 13531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:14.625555 13531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "f270430196cb4484ac1e0eb57fd83b17"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "a70dad9fc0050b97a54bf40d0da226cd"
server_key_iv: "379b4d6914112b6ac83aa9ab33e42ba4"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.626097 13531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "f270430196cb4484ac1e0eb57fd83b17"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "a70dad9fc0050b97a54bf40d0da226cd"
server_key_iv: "379b4d6914112b6ac83aa9ab33e42ba4"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.629595 13531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:14.631968 13546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:14.633189 13531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:58:14.633335 13531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "f270430196cb4484ac1e0eb57fd83b17"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "a70dad9fc0050b97a54bf40d0da226cd"
server_key_iv: "379b4d6914112b6ac83aa9ab33e42ba4"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.633445 13531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:14.647828 13531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:14.650487 13531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:14.650681 13531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:14.651224 13531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:14.652067 13531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:14.652144 13531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:14.652211 13531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:14.652246 13531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:14.660789 13531 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:40267
I20260620 14:58:14.660838 13659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:40267 every 8 connection(s)
I20260620 14:58:14.661664 13531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:14.662113 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 13531
I20260620 14:58:14.662235 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:14.662576 7354 external_mini_cluster.cc:1428] Setting key 8d2787b5ea2f21bd8f61de2727880ce7
WARNING: no policy specified for oryx/127.7.46.130@KRBTEST.COM; defaulting to no policy
Jun 20 14:58:14 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.130@KRBTEST.COM" created.
I20260620 14:58:14.675827 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:14.663494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:48991 (local address 127.7.46.190:36619)
0620 14:58:14.663616 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:14.663625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:14.664635 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 14:58:14.666326 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:14.666330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:14.666390 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:14.666466 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:14.667552 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.668188 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.669285 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.669538 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.669651 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:14.672279 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:14.672282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:14.672284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:14.672307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:14.673643 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:14.674177 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:14.674183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:14.674187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:14.674281 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:14.674606 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:14.674612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:14.674617 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:14.674880 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:14.674988 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:14.675561 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 14:58:14.675659 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:58:14.676870 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:14.663997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.129:48991)
0620 14:58:14.664458 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:14.664514 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:14.666128 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:14.666627 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:14.666634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:14.666974 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:14.667414 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:14.667427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.668311 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.668315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:14.669161 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:14.669170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.669685 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.669689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:14.669848 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:14.670443 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:14.670459 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:14.672052 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:14.673789 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:14.673797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:14.673813 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:14.674075 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:14.674376 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:14.674378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:14.674379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:14.674501 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:14.674982 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:14.674986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:14.675281 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 14:58:14.675515 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 14:58:14.675908 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260620 14:58:14.678030 13660 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36619
I20260620 14:58:14.678290 13660 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:14.678743 13660 heartbeater.cc:511] Master 127.7.46.190:36619 requested a full tablet report, sending...
I20260620 14:58:14.680454 13469 ts_manager.cc:194] Registered new tserver with Master: f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267)
I20260620 14:58:14.682111 13469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:48991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab.
Entry for principal oryx/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab.
I20260620 14:58:14.690356 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36619
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab
--principal=oryx/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:14.800658 13665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:14.800897 13665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:14.800946 13665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:14.804785 13665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:14.804837 13665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:14.804893 13665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:14.809855 13665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab
--principal=oryx/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:14.810768 13665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:14.811687 13665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:14.821321 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:14.821297 13673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:14.821569 13665 server_base.cc:1034] running on GCE node
W20260620 14:58:14.821300 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:14.822064 13665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:14.822691 13665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:14.823835 13665 hybrid_clock.cc:648] HybridClock initialized: now 1781967494823809 us; error 36 us; skew 500 ppm
Jun 20 14:58:14 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:14.826414 13665 init.cc:377] Logged in from keytab as oryx/127.7.46.130@KRBTEST.COM (short username oryx)
I20260620 14:58:14.827276 13665 webserver.cc:460] Webserver started at http://127.7.46.130:42323/ using document root <none> and password file <none>
I20260620 14:58:14.827869 13665 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:14.827965 13665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:14.828187 13665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:14.830116 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "eeb7a459db0242ec83f65b1d7e56b6a4"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "e194ec7ba9d1818ec51e618a99f49460"
server_key_iv: "702c939a4ff8fd214fcbccb8f714bf47"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.830662 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "eeb7a459db0242ec83f65b1d7e56b6a4"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "e194ec7ba9d1818ec51e618a99f49460"
server_key_iv: "702c939a4ff8fd214fcbccb8f714bf47"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.834219 13665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:58:14.836645 13680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:14.837924 13665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:14.838042 13665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "eeb7a459db0242ec83f65b1d7e56b6a4"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "e194ec7ba9d1818ec51e618a99f49460"
server_key_iv: "702c939a4ff8fd214fcbccb8f714bf47"
server_key_version: "encryptionkey@0"
I20260620 14:58:14.838138 13665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:14.856694 13665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:14.859426 13665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:14.859630 13665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:14.860241 13665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:14.861007 13665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:14.861052 13665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:14.861120 13665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:14.861160 13665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:14.869783 13665 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:36791
I20260620 14:58:14.869854 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:36791 every 8 connection(s)
I20260620 14:58:14.870710 13665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:14.876410 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 13665
I20260620 14:58:14.876543 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:14.876843 7354 external_mini_cluster.cc:1428] Setting key cbbec65183fbaba4ef344ba0b3debe4a
Jun 20 14:58:14 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:14.884693 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:14.872611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:35251 (local address 127.7.46.190:36619)
0620 14:58:14.872749 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:14.872753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:14.873540 (+ 787us) server_negotiation.cc:407] Connection header received
0620 14:58:14.874820 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:14.874823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:14.874879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:14.874963 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:14.876174 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.876953 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.878256 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.878476 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.878599 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:14.881245 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:14.881248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:14.881256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:14.881284 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:14.882670 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:14.883186 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:14.883190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:14.883192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:14.883269 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:14.883604 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:14.883609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:14.883610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:14.883826 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:14.883915 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:14.884403 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 14:58:14.884526 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:58:14.885502 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:14.872972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.130:35251)
0620 14:58:14.873387 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:14.873427 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:14.874606 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:14.875156 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:14.875163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:14.875496 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:14.876026 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:14.876040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.877082 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.877085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:14.878139 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:14.878149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:14.878633 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:14.878638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:14.878803 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:14.879383 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:14.879399 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:14.881086 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:14.882804 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:14.882810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:14.882823 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:14.883081 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:14.883361 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:14.883366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:14.883368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:14.883503 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:14.883924 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:14.883932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:14.884172 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 14:58:14.884362 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 14:58:14.884663 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.7.46.131@KRBTEST.COM; defaulting to no policy
I20260620 14:58:14.886636 13794 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36619
I20260620 14:58:14.886865 13794 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:14.887319 13794 heartbeater.cc:511] Master 127.7.46.190:36619 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.131@KRBTEST.COM" created.
I20260620 14:58:14.888569 13469 ts_manager.cc:194] Registered new tserver with Master: eeb7a459db0242ec83f65b1d7e56b6a4 (127.7.46.130:36791)
I20260620 14:58:14.889389 13469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:35251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab.
Entry for principal oryx/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab.
I20260620 14:58:14.902786 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36619
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab
--principal=oryx/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:15.011324 13799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:15.011607 13799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:15.011682 13799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:15.015694 13799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:15.015784 13799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:15.015918 13799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:15.021157 13799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab
--principal=oryx/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:15.022189 13799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:15.023193 13799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:15.032660 13807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:15.032684 13805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.032728 13799 server_base.cc:1034] running on GCE node
W20260620 14:58:15.032682 13804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.033224 13799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:15.033833 13799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:15.035027 13799 hybrid_clock.cc:648] HybridClock initialized: now 1781967495034997 us; error 48 us; skew 500 ppm
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:15.037683 13799 init.cc:377] Logged in from keytab as oryx/127.7.46.131@KRBTEST.COM (short username oryx)
I20260620 14:58:15.038599 13799 webserver.cc:460] Webserver started at http://127.7.46.131:42167/ using document root <none> and password file <none>
I20260620 14:58:15.039148 13799 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:15.039237 13799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:15.039456 13799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:15.041544 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "ad147cf6977d42b794ff577c738f277a"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "f47d53d134def574025474261d191af6"
server_key_iv: "89c6f9efc1c542fc01ed93c47f7ce0c5"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.042124 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "ad147cf6977d42b794ff577c738f277a"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "f47d53d134def574025474261d191af6"
server_key_iv: "89c6f9efc1c542fc01ed93c47f7ce0c5"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.045809 13799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:15.048147 13814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.049311 13799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:15.049445 13799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "ad147cf6977d42b794ff577c738f277a"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "f47d53d134def574025474261d191af6"
server_key_iv: "89c6f9efc1c542fc01ed93c47f7ce0c5"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.049551 13799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:15.072196 13799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:15.074934 13799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:15.075134 13799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:15.075706 13799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:15.076546 13799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:15.076617 13799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.076684 13799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:15.076717 13799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.085448 13799 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:42801
I20260620 14:58:15.085467 13927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:42801 every 8 connection(s)
I20260620 14:58:15.086292 13799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:15.088848 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 13799
I20260620 14:58:15.088970 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:15.089316 7354 external_mini_cluster.cc:1428] Setting key de5779fb1ef4df5e287e5e0c373330dc
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:15.099977 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.088078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:37069 (local address 127.7.46.190:36619)
0620 14:58:15.088223 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.088227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.089229 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 14:58:15.090806 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.090808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.090863 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.090948 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.092159 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.092675 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.093822 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.094038 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.094116 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.096720 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.096724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.096727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.096756 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.097991 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.098485 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.098490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.098492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.098571 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.098899 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.098902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.098904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.099085 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.099191 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.099661 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.099808 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:58:15.100697 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.088507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.131:37069)
0620 14:58:15.089080 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.089123 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.090556 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.091106 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.091112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.091551 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.092039 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.092052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.092775 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.092777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.093723 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.093732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.094159 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.094162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.094392 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.094997 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.095013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.096564 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.098111 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.098117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.098131 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.098379 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.098665 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.098669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.098671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.098803 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.099198 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.099207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.099432 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.099618 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.099945 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":148,"threads_started":1}
I20260620 14:58:15.101724 13928 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36619
I20260620 14:58:15.102005 13928 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:15.102525 13928 heartbeater.cc:511] Master 127.7.46.190:36619 requested a full tablet report, sending...
I20260620 14:58:15.103539 13469 ts_manager.cc:194] Registered new tserver with Master: ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801)
I20260620 14:58:15.104166 13469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:37069
I20260620 14:58:15.114679 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.46.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.189.keytab.
Entry for principal oryx/127.7.46.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.189.keytab.
I20260620 14:58:15.140419 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.189:37881
--webserver_interface=127.7.46.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.189.keytab
--principal=oryx/127.7.46.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:36619,127.7.46.189:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:15.249547 13933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:15.249778 13933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:15.249861 13933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:15.253873 13933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:15.253921 13933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:15.253939 13933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:15.253993 13933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:15.254017 13933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:15.259464 13933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:36619,127.7.46.189:37881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.189.keytab
--principal=oryx/127.7.46.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.189:37881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.46.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
W20260620 14:58:15.259678 13933 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.46.190:36619,127.7.46.189:37881'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:58:15.260753 13933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:15.261893 13933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:15.269958 13939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.269979 13933 server_base.cc:1034] running on GCE node
W20260620 14:58:15.269927 13938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:15.269908 13941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.270793 13933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:15.271878 13933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:15.273032 13933 hybrid_clock.cc:648] HybridClock initialized: now 1781967495273003 us; error 41 us; skew 500 ppm
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:15.275679 13933 init.cc:377] Logged in from keytab as oryx/127.7.46.189@KRBTEST.COM (short username oryx)
I20260620 14:58:15.276515 13933 webserver.cc:460] Webserver started at http://127.7.46.189:34301/ using document root <none> and password file <none>
I20260620 14:58:15.277045 13933 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:15.277104 13933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:15.277361 13933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:15.279300 13933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data/instance:
uuid: "65ffb39156b04715b3ba9b10e07387e9"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "b651e769be4243b43ee10aecb7facd37"
server_key_iv: "81334efd311d3dd2e61554e6500eb0e1"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.279883 13933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal/instance:
uuid: "65ffb39156b04715b3ba9b10e07387e9"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "b651e769be4243b43ee10aecb7facd37"
server_key_iv: "81334efd311d3dd2e61554e6500eb0e1"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.283433 13933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:58:15.285812 13948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.286917 13933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:15.287037 13933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
uuid: "65ffb39156b04715b3ba9b10e07387e9"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "b651e769be4243b43ee10aecb7facd37"
server_key_iv: "81334efd311d3dd2e61554e6500eb0e1"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.287133 13933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:15.299378 13933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:15.301997 13933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:15.302187 13933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:15.308632 13933 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.189:37881
I20260620 14:58:15.308905 14000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.189:37881 every 8 connection(s)
I20260620 14:58:15.309553 13933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
I20260620 14:58:15.312152 14001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:15.315551 14001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:58:15.316025 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 13933
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.189@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.46.189@KRBTEST.COM
I20260620 14:58:15.328528 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.317417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.7.46.190:36619)
0620 14:58:15.317564 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.317571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.318061 (+ 490us) server_negotiation.cc:407] Connection header received
0620 14:58:15.318783 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.318786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.318846 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.318947 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.319868 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.320367 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.321449 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.321734 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.321810 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.324466 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.324468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.324470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.324494 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.326182 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.326739 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.326745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.326749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.326836 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.327199 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.327202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.327203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.327471 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.327589 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.328109 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.328238 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:58:15.329157 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.317430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:59920)
0620 14:58:15.317929 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.317964 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.318596 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.319028 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.319033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.319432 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.319726 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.319741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.320499 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.320502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.321334 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.321343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.321860 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.321864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.322032 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.322677 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.322701 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.324284 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.326285 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.326292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.326299 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.326636 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.326933 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.326936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.326938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.327106 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.327598 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.327602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.327867 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.328045 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.328345 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"spinlock_wait_cycles":15616,"thread_start_us":172,"threads_started":1}
I20260620 14:58:15.329164 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.317426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.7.46.189:37881)
0620 14:58:15.317821 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.317838 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.317875 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:58:15.318583 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.318601 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.318781 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.318943 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.319677 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.320723 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.321610 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.322041 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.322140 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.324899 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.324909 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.324918 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.324958 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.326704 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.327199 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.327204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.327206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.327286 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.327547 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.327552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.327557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.327872 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.328038 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.328297 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.328419 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260620 14:58:15.330467 14001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.189@KRBTEST.COM for oryx/127.7.46.189@KRBTEST.COM
I20260620 14:58:15.338478 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.330961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.0.0.1:35466)
0620 14:58:15.331068 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.331081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.331143 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.331380 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.331383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.331535 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.331611 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.331615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.332354 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.332357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.332860 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.332867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.333777 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.333780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.333913 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.334296 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.334306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.335435 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.337203 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.337206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.337207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.337440 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.337718 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.337721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.337722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.337795 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.338104 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.338106 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.338173 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.338250 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.338337 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":13}
I20260620 14:58:15.338502 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.331031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.7.46.189:37881)
0620 14:58:15.331145 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.331151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.331167 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:15.331235 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.331237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.331282 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.331358 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.331720 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.332239 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.333016 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.333663 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.333780 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.335599 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.335602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.335604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.335625 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.337090 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.337557 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.337559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.337561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.337639 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.337871 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.337873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.337874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.338027 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.338099 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.338303 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.338384 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":40}
I20260620 14:58:15.341431 14001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Bootstrap starting.
I20260620 14:58:15.343693 14001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:15.344527 14001 log.cc:824] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:15.346300 14001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: No bootstrap required, opened a new log
I20260620 14:58:15.349134 14001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } }
I20260620 14:58:15.349319 14001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:15.349352 14001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ffb39156b04715b3ba9b10e07387e9, State: Initialized, Role: FOLLOWER
I20260620 14:58:15.349701 14001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } }
I20260620 14:58:15.350085 14007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } } }
I20260620 14:58:15.350191 14007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:58:15.350764 14001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:58:15.355420 14018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:58:15.355521 14018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:58:15.357389 13933 master_runner.cc:428] Detected that this master 65ffb39156b04715b3ba9b10e07387e9 is joining an existing cluster
I20260620 14:58:15.357445 13933 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.46.189:37881
I20260620 14:58:15.367625 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.359694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.7.46.189:37881)
0620 14:58:15.359820 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.359825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.359936 (+ 111us) server_negotiation.cc:407] Connection header received
0620 14:58:15.360033 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.360035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.360076 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.360150 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.360676 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.361410 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.362606 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.362803 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.362921 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.364186 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.364191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.364193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.364218 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.365836 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.366285 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.366288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.366290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.366365 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.366665 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.366669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.366673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.366882 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.366981 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.367316 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.367437 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":39}
I20260620 14:58:15.368031 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.359346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.0.0.1:35482)
0620 14:58:15.359833 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.359846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.359952 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.360197 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.360200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.360441 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.360567 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.360575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.361540 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.361544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.362490 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.362501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.362929 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.362932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.363080 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.363591 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.363604 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.364054 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.365959 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.365963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.365965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.366194 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.366457 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.366461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.366463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.366559 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.367029 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.367033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.367131 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.367254 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.367818 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":31,"thread_start_us":88,"threads_started":1}
I20260620 14:58:15.369609 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.359316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:59928)
0620 14:58:15.359580 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.359596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.359700 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.360131 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.360137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.360364 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.360452 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.360458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.362632 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.362635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.363542 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.363552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.364490 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.364494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.364592 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.365442 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.365457 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.366113 (+ 656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.368205 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.368210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.368212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.368435 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.368738 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.368740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.368742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.368840 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.369251 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.369254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.369320 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.369401 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.369488 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260620 14:58:15.369861 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.359370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.7.46.190:36619)
0620 14:58:15.359662 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.359667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.359683 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:15.359853 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.359857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.359933 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.359987 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.360584 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.361414 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.363671 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.364054 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.364165 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.366257 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.366262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.366267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.366295 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.368078 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.368539 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.368543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.368545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.368633 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.368934 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.368937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.368939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.369150 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.369246 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.369480 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.369592 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.190@KRBTEST.COM for oryx/127.7.46.189@KRBTEST.COM
I20260620 14:58:15.384649 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.373565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.0.0.1:35484)
0620 14:58:15.376119 (+ 2554us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.376155 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.376296 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.376605 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.376611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.376929 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.377039 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.377046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.377738 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.377742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.378350 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.378358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.378801 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.378804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.378919 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.379429 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.379441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.380780 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.382662 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.382671 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.382676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.382953 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.383271 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.383276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.383278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.383370 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.383882 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.383891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.384040 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.384190 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.384395 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2382,"thread_start_us":84,"threads_started":1}
I20260620 14:58:15.385187 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.373583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.7.46.189:37881)
0620 14:58:15.373686 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.373690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.376230 (+ 2540us) server_negotiation.cc:407] Connection header received
0620 14:58:15.376428 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.376430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.376476 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.376573 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.377153 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.377629 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.378468 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.378691 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.378799 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.380911 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.380914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.380917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.380945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.382550 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.383082 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.383085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.383087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.383168 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.383469 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.383475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.383479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.383703 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.383825 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.385015 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.385096 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:58:15.386600 13469 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.46.189:37881: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9"
member_type: NON_VOTER
last_known_addr {
host: "127.7.46.189"
port: 37881
}
attrs {
promote: true
}
}
dest_uuid: "6496092390a24fef98320c7d834845bc"
cas_config_opid_index: -1
I20260620 14:58:15.387241 13469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } }
I20260620 14:58:15.389019 14028 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 65ffb39156b04715b3ba9b10e07387e9 (127.7.46.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } }
I20260620 14:58:15.389604 13975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:15.390771 14028 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.46.189:37881
I20260620 14:58:15.391178 13933 master.cc:537] Master@127.7.46.189:37881 shutting down...
I20260620 14:58:15.391307 13975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Refusing update from remote peer 6496092390a24fef98320c7d834845bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:58:15.391695 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } }
I20260620 14:58:15.391812 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:15.392031 14028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:58:15.393743 13453 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 14:58:15.396076 13933 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:15.396425 13933 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:15.396612 13933 tablet_replica.cc:330] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: stopping tablet replica
W20260620 14:58:15.396661 13453 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc -> Peer 65ffb39156b04715b3ba9b10e07387e9 (127.7.46.189:37881): Couldn't send request to peer 65ffb39156b04715b3ba9b10e07387e9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.7.46.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.188@KRBTEST.COM" created.
I20260620 14:58:15.401018 13933 master.cc:552] Master@127.7.46.189:37881 shutdown complete.
I20260620 14:58:15.401074 13933 master_runner.cc:305] Clearing existing system tablet
I20260620 14:58:15.401321 13933 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:15.401382 13933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:15.402777 13933 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:58:15.403579 14034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.407963 13933 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:15.408113 13933 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 14:58:15.408159 13933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
uuid: "65ffb39156b04715b3ba9b10e07387e9"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "b651e769be4243b43ee10aecb7facd37"
server_key_iv: "81334efd311d3dd2e61554e6500eb0e1"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.409832 13933 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:15.412472 13933 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:58:15.412609 13933 log.cc:1196] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:58:15.413220 13933 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Deleting consensus metadata
I20260620 14:58:15.413401 13933 master_runner.cc:315] Copying system tablet from 127.7.46.190:36619
I20260620 14:58:15.414698 13933 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: tablet copy: Beginning tablet copy session from remote peer at address 127.7.46.190:36619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.188.keytab.
Entry for principal oryx/127.7.46.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.188.keytab.
I20260620 14:58:15.416545 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.188:33569
--webserver_interface=127.7.46.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.188.keytab
--principal=oryx/127.7.46.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:36619,127.7.46.189:37881,127.7.46.188:33569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:58:15.423374 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.415323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.7.46.190:36619)
0620 14:58:15.415462 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.415467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.415696 (+ 229us) server_negotiation.cc:407] Connection header received
0620 14:58:15.416002 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.416006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.416057 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.416140 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.416924 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.417444 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.418761 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.419115 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.419226 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.420284 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.420289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.420292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.420318 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.421875 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.422335 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.422338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.422339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.422414 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.422678 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.422680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.422681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.422826 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.422898 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.423175 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.423264 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:58:15.423439 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.415266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:59934)
0620 14:58:15.415591 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.415623 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.415771 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.416157 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.416159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.416351 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.416772 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.416782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.417567 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.417569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.418614 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.418628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.419234 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.419237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.419306 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.419851 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.419863 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.420156 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.421997 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.422000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.422002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.422243 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.422506 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.422509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.422511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.422598 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.422912 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.422915 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.423015 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.423123 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.423249 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":96,"threads_started":1}
I20260620 14:58:15.423925 13489 tablet_copy_service.cc:140] P 6496092390a24fef98320c7d834845bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 65ffb39156b04715b3ba9b10e07387e9 ({username='oryx', principal='oryx/127.7.46.189@KRBTEST.COM'} at 127.0.0.1:59934)
I20260620 14:58:15.424103 13489 tablet_copy_service.cc:161] P 6496092390a24fef98320c7d834845bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 65ffb39156b04715b3ba9b10e07387e9 at {username='oryx', principal='oryx/127.7.46.189@KRBTEST.COM'} at 127.0.0.1:59934: session id = 65ffb39156b04715b3ba9b10e07387e9-00000000000000000000000000000000
I20260620 14:58:15.425485 13489 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:58:15.426791 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:58:15.428544 13933 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: tablet copy: Starting download of 0 data blocks...
I20260620 14:58:15.428740 13933 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: tablet copy: Starting download of 1 WAL segments...
I20260620 14:58:15.430235 13933 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:58:15.431635 13489 tablet_copy_service.cc:342] P 6496092390a24fef98320c7d834845bc: Request end of tablet copy session 65ffb39156b04715b3ba9b10e07387e9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.46.189@KRBTEST.COM'} at 127.0.0.1:59934
I20260620 14:58:15.431751 13489 tablet_copy_service.cc:433] P 6496092390a24fef98320c7d834845bc: ending tablet copy session 65ffb39156b04715b3ba9b10e07387e9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 65ffb39156b04715b3ba9b10e07387e9
I20260620 14:58:15.438531 13933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:15.440197 14042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:15.440173 14045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.440605 13933 server_base.cc:1034] running on GCE node
W20260620 14:58:15.443890 14043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.444160 13933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:15.444474 13933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:15.445636 13933 hybrid_clock.cc:648] HybridClock initialized: now 1781967495445625 us; error 24 us; skew 500 ppm
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:15.447052 13933 init.cc:377] Logged in from keytab as oryx/127.7.46.189@KRBTEST.COM (short username oryx)
I20260620 14:58:15.447502 13933 webserver.cc:460] Webserver started at http://127.7.46.189:37481/ using document root <none> and password file <none>
I20260620 14:58:15.447686 13933 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:15.447741 13933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:15.448834 13933 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:15.449728 14052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.449950 13933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:15.450037 13933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
uuid: "65ffb39156b04715b3ba9b10e07387e9"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "b651e769be4243b43ee10aecb7facd37"
server_key_iv: "81334efd311d3dd2e61554e6500eb0e1"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.450116 13933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:15.456972 13933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:15.458940 13933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:15.463915 13933 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.189:37881
I20260620 14:58:15.464560 13933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
I20260620 14:58:15.466075 14105 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:58:15.466859 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Bootstrap starting.
I20260620 14:58:15.469782 14104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.189:37881 every 8 connection(s)
I20260620 14:58:15.474964 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:58:15.475274 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: Bootstrap complete.
I20260620 14:58:15.475759 14105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } }
I20260620 14:58:15.475965 14105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65ffb39156b04715b3ba9b10e07387e9, State: Initialized, Role: LEARNER
I20260620 14:58:15.476080 14105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } }
I20260620 14:58:15.476423 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } }
I20260620 14:58:15.476562 14105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:15.476588 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:58:15.479509 14119 catalog_manager.cc:1259] Loaded cluster ID: 7c740eac043144dc94b89e76b94c04f0
I20260620 14:58:15.479585 14119 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: loading cluster ID for follower catalog manager: success
I20260620 14:58:15.481137 14119 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: acquiring CA information for follower catalog manager: success
I20260620 14:58:15.482218 14119 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:58:15.572420 14041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:15.572664 14041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:15.572705 14041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:15.576555 14041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:15.576601 14041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:15.576619 14041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:15.576637 14041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:15.576651 14041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:15.581619 14041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:36619,127.7.46.189:37881,127.7.46.188:33569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.188.keytab
--principal=oryx/127.7.46.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.188:33569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.46.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:15.582516 14041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:15.583459 14041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:15.591567 14125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:15.591562 14127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:15.591563 14124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.591806 14041 server_base.cc:1034] running on GCE node
I20260620 14:58:15.592289 14041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:15.593338 14041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:15.594583 14041 hybrid_clock.cc:648] HybridClock initialized: now 1781967495594560 us; error 58 us; skew 500 ppm
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:15.597389 14041 init.cc:377] Logged in from keytab as oryx/127.7.46.188@KRBTEST.COM (short username oryx)
I20260620 14:58:15.598268 14041 webserver.cc:460] Webserver started at http://127.7.46.188:36891/ using document root <none> and password file <none>
I20260620 14:58:15.598800 14041 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:15.598860 14041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:15.599086 14041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:15.601135 14041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data/instance:
uuid: "d00d1e51f2c04f64b06a066b087582d0"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "5f9942000e4d3e32f05a27993d6cf738"
server_key_iv: "7e9ba4ca3fe63eaada8a7e40b5e01998"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.601722 14041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal/instance:
uuid: "d00d1e51f2c04f64b06a066b087582d0"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "5f9942000e4d3e32f05a27993d6cf738"
server_key_iv: "7e9ba4ca3fe63eaada8a7e40b5e01998"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.605289 14041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:15.607448 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.608675 14041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:15.608819 14041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
uuid: "d00d1e51f2c04f64b06a066b087582d0"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "5f9942000e4d3e32f05a27993d6cf738"
server_key_iv: "7e9ba4ca3fe63eaada8a7e40b5e01998"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.608924 14041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:15.637971 14041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:15.640712 14041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:15.640913 14041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:15.648152 14041 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.188:33569
I20260620 14:58:15.648427 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.188:33569 every 8 connection(s)
I20260620 14:58:15.649171 14041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
I20260620 14:58:15.651953 14187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:15.655622 14187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:58:15.658418 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 14041
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.188@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.46.188@KRBTEST.COM
I20260620 14:58:15.669432 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.657676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.7.46.190:36619)
0620 14:58:15.657831 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.657837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.658361 (+ 524us) server_negotiation.cc:407] Connection header received
0620 14:58:15.659349 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.659352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.659409 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.659493 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.660808 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.661567 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.662993 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.663320 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.663407 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.665736 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.665740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.665743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.665772 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.667361 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.668009 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.668012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.668013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.668083 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.668430 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.668434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.668440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.668602 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.668708 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.669162 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.669272 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:58:15.670215 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.657791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:59938)
0620 14:58:15.658223 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.658263 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.659146 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.659677 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.659683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.660127 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.660605 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.660624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.661720 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.661723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.662859 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.662871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.663558 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.663561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.663667 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.664226 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.664241 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.665593 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.667466 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.667473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.667482 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.667893 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.668190 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.668193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.668196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.668328 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.668704 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.668709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.668934 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.669106 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.669381 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":26,"spinlock_wait_cycles":768,"thread_start_us":100,"threads_started":1}
I20260620 14:58:15.670218 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.659224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.7.46.188:33569)
0620 14:58:15.659565 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.659576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.659597 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:58:15.659655 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.659659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.659870 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.660011 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.660835 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.661955 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.662740 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.663227 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.663389 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.666177 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.666183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.666198 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.666218 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.668012 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.668409 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.668415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.668417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.668497 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.668764 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.668769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.668770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.668995 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.669201 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.669421 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.669569 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":256,"spinlock_wait_cycles":5248,"thread_start_us":146,"threads_started":1}
I20260620 14:58:15.671291 14187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.188@KRBTEST.COM for oryx/127.7.46.189@KRBTEST.COM
I20260620 14:58:15.678696 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.671741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.0.0.1:35492)
0620 14:58:15.671887 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.671897 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.671949 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.672346 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.672348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.672462 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.672542 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.672547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.673348 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.673350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.673847 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.673854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.674241 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.674244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.674309 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.674735 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.674752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.675892 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.677430 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.677433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.677434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.677659 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.677908 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.677910 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.677911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.677983 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.678303 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.678305 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.678370 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.678505 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.678597 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 14:58:15.678789 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.671865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.7.46.189:37881)
0620 14:58:15.672131 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.672136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.672151 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:58:15.672194 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.672197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.672244 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.672317 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.672635 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.673263 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.673935 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.674147 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.674219 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.676006 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.676009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.676011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.676030 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.677299 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.677753 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.677755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.677757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.677828 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.678076 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.678078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.678079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.678228 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.678299 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.678549 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.678637 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260620 14:58:15.679440 14187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.188@KRBTEST.COM for oryx/127.7.46.188@KRBTEST.COM
I20260620 14:58:15.684407 13660 heartbeater.cc:503] Master 127.7.46.190:36619 was elected leader, sending a full tablet report...
I20260620 14:58:15.687124 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.680002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.7.46.188:33569)
0620 14:58:15.680097 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.680104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.680118 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:15.680149 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.680152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.680187 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.680266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.680589 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.681066 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.681770 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.682433 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.682547 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.684213 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.684215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.684217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.684254 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.685645 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.686060 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.686063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.686065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.686132 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.686392 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.686394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.686396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.686545 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.686647 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.686867 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.686995 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":40}
I20260620 14:58:15.687124 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.679887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:33569 (local address 127.0.0.1:39058)
0620 14:58:15.679974 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.679985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.680071 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.680275 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.680278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.680427 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.680479 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.680484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.681182 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.681185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.681661 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.681668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.682547 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.682550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.682625 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.682984 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.682997 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.684099 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.685771 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.685774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.685776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.685953 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.686221 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.686225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.686227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.686301 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.686638 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.686641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.686715 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.686826 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.686946 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:58:15.689760 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Bootstrap starting.
I20260620 14:58:15.691967 14187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:15.692662 14187 log.cc:824] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:15.694468 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: No bootstrap required, opened a new log
I20260620 14:58:15.697461 14187 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } }
I20260620 14:58:15.697642 14187 raft_consensus.cc:374] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:15.697723 14187 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00d1e51f2c04f64b06a066b087582d0, State: Initialized, Role: FOLLOWER
I20260620 14:58:15.698119 14187 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } }
I20260620 14:58:15.698565 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } } }
I20260620 14:58:15.698697 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:58:15.699242 14187 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:58:15.703611 14204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:58:15.703692 14204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:58:15.706614 14041 master_runner.cc:428] Detected that this master d00d1e51f2c04f64b06a066b087582d0 is joining an existing cluster
I20260620 14:58:15.706676 14041 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.46.188:33569
I20260620 14:58:15.718137 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.709411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.7.46.190:36619)
0620 14:58:15.709548 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.709554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.709572 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:58:15.709615 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.709621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.709669 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.709749 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.710455 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.711281 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.712594 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.712930 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.713018 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.715087 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.715090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.715092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.715114 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.716606 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.717053 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.717056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.717057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.717126 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.717438 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.717440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.717441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.717607 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.717684 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.717938 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.718025 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:15.718201 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.708873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:59944)
0620 14:58:15.709458 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.709470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.709540 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.709891 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.709896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.710156 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.710300 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.710309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.711417 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.711420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.712442 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.712453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.713216 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.713219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.713311 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.714408 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.714421 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.714945 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.716708 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.716711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.716713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.716961 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.717214 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.717217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.717219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.717338 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.717701 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.717703 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.717789 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.717893 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.718024 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":28,"thread_start_us":93,"threads_started":1}
I20260620 14:58:15.723237 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 13531
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967494, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.190@KRBTEST.COM for oryx/127.7.46.188@KRBTEST.COM
I20260620 14:58:15.730233 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:40267
--local_ip_for_outbound_sockets=127.7.46.129
--tserver_master_addrs=127.7.46.190:36619
--webserver_port=38931
--webserver_interface=127.7.46.129
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab
--principal=oryx/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881,127.7.46.188:33569 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:58:15.740765 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.721687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:33569 (local address 127.0.0.1:39076)
0620 14:58:15.721802 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.721819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.721898 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.722282 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.722285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.722464 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.722548 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.722554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.723558 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.723560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.724384 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.724394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.725004 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.725019 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.725155 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.725648 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.725667 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.727202 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.739141 (+ 11939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.739144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.739146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.739401 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.739708 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.739711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.739713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.739824 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.740265 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.740268 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.740364 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.740443 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.740593 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:58:15.741573 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.708873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.0.0.1:35500)
0620 14:58:15.709178 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.709194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.709285 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.710328 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.710333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.710546 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.710629 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.710635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.711681 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.711684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.713137 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.713146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.714066 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.714069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.714220 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.736831 (+ 22611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.736854 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.737442 (+ 588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.739696 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.739702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.739705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.740024 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.740378 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.740383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.740386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.740484 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.741030 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.741033 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.741125 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.741211 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.741373 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":104,"thread_start_us":113,"threads_started":1}
I20260620 14:58:15.742177 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.709691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.7.46.189:37881)
0620 14:58:15.710037 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.710043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.710056 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:58:15.710095 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.710098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.710144 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.710220 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.710753 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.711541 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.713269 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.713939 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.714068 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.737666 (+ 23598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.737673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.737679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.737717 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.739546 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.740158 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.740164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.740169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.740260 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.740610 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.740613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.740615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.740865 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.740970 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.741890 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.742029 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260620 14:58:15.743475 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.721740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.7.46.188:33569)
0620 14:58:15.722007 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.722014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.722027 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:58:15.722086 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.722089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.722133 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.722221 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.722669 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.723441 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.724526 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.724873 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.724980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.727360 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.727367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.727372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.727402 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.739006 (+ 11604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.739523 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.739526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.739528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.739609 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.739933 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.739947 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.739949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.740157 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.743081 (+ 2924us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.743153 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.743315 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
I20260620 14:58:15.744453 13469 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.46.188:33569: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0"
member_type: NON_VOTER
last_known_addr {
host: "127.7.46.188"
port: 33569
}
attrs {
promote: true
}
}
dest_uuid: "6496092390a24fef98320c7d834845bc"
cas_config_opid_index: 5
I20260620 14:58:15.745184 13469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } }
I20260620 14:58:15.746047 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.709841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.7.46.188:33569)
0620 14:58:15.709933 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.709939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.710959 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 14:58:15.711088 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.711091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.711120 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.711160 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.711691 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.712509 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.713795 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.714178 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.714281 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.741909 (+ 27628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.741912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.741915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.741948 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.744183 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.744611 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.744614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.744616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.744698 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.745021 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.745024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.745026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.745219 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.745313 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.745670 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.745775 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":38}
I20260620 14:58:15.746466 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.709416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:33569 (local address 127.0.0.1:39072)
0620 14:58:15.710889 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.710902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.711006 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.711252 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.711255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.711441 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.711554 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.711560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.712765 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.712768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.713663 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.713675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.714291 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.714296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.714382 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.738065 (+ 23683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.738085 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.741580 (+ 3495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.744306 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.744308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.744310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.744527 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.744795 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.744798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.744800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.744884 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.745348 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.745351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.745480 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.746169 (+ 689us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.746296 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"spinlock_wait_cycles":896,"thread_start_us":117,"threads_started":1}
I20260620 14:58:15.747618 14161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:15.747747 14029 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d00d1e51f2c04f64b06a066b087582d0 (127.7.46.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.749159 14161 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 FOLLOWER]: Refusing update from remote peer 6496092390a24fef98320c7d834845bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:58:15.749380 14029 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.46.188:33569
I20260620 14:58:15.749773 14029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.749905 14029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:15.750012 14029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:15.750972 14041 master.cc:537] Master@127.7.46.188:33569 shutting down...
I20260620 14:58:15.755366 14161 raft_consensus.cc:858] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } }
W20260620 14:58:15.755584 14161 raft_consensus.cc:1565] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } }
I20260620 14:58:15.756742 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.747262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.0.0.1:35514)
0620 14:58:15.752152 (+ 4890us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.752167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.752245 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.752855 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.752858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.752878 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:15.752986 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.753001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.754115 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.754119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.755297 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.755310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.756351 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.756355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.756441 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.756488 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.756554 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.756619 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4820}
I20260620 14:58:15.757386 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.747458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.7.46.189:37881)
0620 14:58:15.752433 (+ 4975us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.752437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.752451 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:15.752495 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.752500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.752597 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.753107 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:15.753220 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.753986 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.755518 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.756212 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.757010 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.757162 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.757202 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.757266 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4890}
I20260620 14:58:15.757296 14161 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 65ffb39156b04715b3ba9b10e07387e9 (127.7.46.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } }
I20260620 14:58:15.757809 14079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 LEARNER]: Refusing update from remote peer 6496092390a24fef98320c7d834845bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:58:15.758244 14028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:58:15.759366 14219 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } }
I20260620 14:58:15.759518 14219 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:58:15.759666 14219 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } }
I20260620 14:58:15.759814 14219 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:58:15.760094 14079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d00d1e51f2c04f64b06a066b087582d0 (127.7.46.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.761909 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6496092390a24fef98320c7d834845bc. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.762027 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:58:15.761991 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.762116 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:15.762109 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.762184 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:58:15.762953 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6496092390a24fef98320c7d834845bc. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: NON_VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: true } } }
I20260620 14:58:15.763065 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260620 14:58:15.763404 13455 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc -> Peer d00d1e51f2c04f64b06a066b087582d0 (127.7.46.188:33569): Couldn't send request to peer d00d1e51f2c04f64b06a066b087582d0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:58:15.765108 14029 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: attempting to promote NON_VOTER 65ffb39156b04715b3ba9b10e07387e9 to VOTER
I20260620 14:58:15.765400 14041 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 14:58:15.765604 14041 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 14:58:15.765542 14029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } }
I20260620 14:58:15.765677 14041 tablet_replica.cc:330] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: stopping tablet replica
W20260620 14:58:15.767359 13455 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc -> Peer d00d1e51f2c04f64b06a066b087582d0 (127.7.46.188:33569): Couldn't send request to peer d00d1e51f2c04f64b06a066b087582d0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:58:15.768071 14079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 LEARNER]: Refusing update from remote peer 6496092390a24fef98320c7d834845bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:58:15.768483 14028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:58:15.770146 14029 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 65ffb39156b04715b3ba9b10e07387e9 (127.7.46.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.771463 14029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.771555 14029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:15.771669 14029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.771733 14029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:15.772485 14079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 65ffb39156b04715b3ba9b10e07387e9 (127.7.46.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.773931 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.774071 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:58:15.787732 14041 master.cc:552] Master@127.7.46.188:33569 shutdown complete.
I20260620 14:58:15.787813 14041 master_runner.cc:305] Clearing existing system tablet
I20260620 14:58:15.788065 14041 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:15.788151 14041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:15.789206 14041 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:15.790048 14226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.790282 14041 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:15.790432 14041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:15.790469 14041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
uuid: "d00d1e51f2c04f64b06a066b087582d0"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "5f9942000e4d3e32f05a27993d6cf738"
server_key_iv: "7e9ba4ca3fe63eaada8a7e40b5e01998"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.791903 14041 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:15.793859 14041 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:58:15.793949 14041 log.cc:1196] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:58:15.794220 14041 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Deleting consensus metadata
I20260620 14:58:15.794402 14041 master_runner.cc:315] Copying system tablet from 127.7.46.190:36619
I20260620 14:58:15.795620 14041 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: tablet copy: Beginning tablet copy session from remote peer at address 127.7.46.190:36619
I20260620 14:58:15.804035 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.796250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.7.46.190:36619)
0620 14:58:15.796386 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.796392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.796578 (+ 186us) server_negotiation.cc:407] Connection header received
0620 14:58:15.796723 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.796726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.796778 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.796867 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.797306 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.797831 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.799081 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.799372 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.799487 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.800730 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.800734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.800737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.800780 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.802335 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.802785 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.802789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.802790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.802874 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.803161 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.803165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.803167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.803348 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.803456 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.803724 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.803879 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:58:15.804044 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.796186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:59948)
0620 14:58:15.796472 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.796492 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.796626 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.796881 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.796884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.797086 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.797181 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.797187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.797943 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.797948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.798952 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.798962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.799493 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.799496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.799599 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.800181 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.800195 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.800588 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.802468 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.802471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.802474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.802681 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.802973 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.802976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.802978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.803068 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.803445 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.803448 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.803557 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.803663 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.803827 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
I20260620 14:58:15.804906 13489 tablet_copy_service.cc:140] P 6496092390a24fef98320c7d834845bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d00d1e51f2c04f64b06a066b087582d0 ({username='oryx', principal='oryx/127.7.46.188@KRBTEST.COM'} at 127.0.0.1:59948)
I20260620 14:58:15.805029 13489 tablet_copy_service.cc:161] P 6496092390a24fef98320c7d834845bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d00d1e51f2c04f64b06a066b087582d0 at {username='oryx', principal='oryx/127.7.46.188@KRBTEST.COM'} at 127.0.0.1:59948: session id = d00d1e51f2c04f64b06a066b087582d0-00000000000000000000000000000000
I20260620 14:58:15.805877 13489 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:58:15.807060 14041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:58:15.809077 14041 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: tablet copy: Starting download of 0 data blocks...
I20260620 14:58:15.809278 14041 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: tablet copy: Starting download of 1 WAL segments...
I20260620 14:58:15.810636 14041 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:58:15.812031 13489 tablet_copy_service.cc:342] P 6496092390a24fef98320c7d834845bc: Request end of tablet copy session d00d1e51f2c04f64b06a066b087582d0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.46.188@KRBTEST.COM'} at 127.0.0.1:59948
I20260620 14:58:15.812124 13489 tablet_copy_service.cc:433] P 6496092390a24fef98320c7d834845bc: ending tablet copy session d00d1e51f2c04f64b06a066b087582d0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d00d1e51f2c04f64b06a066b087582d0
I20260620 14:58:15.818699 14041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:15.820288 14233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.820640 14041 server_base.cc:1034] running on GCE node
W20260620 14:58:15.820791 14236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:15.824252 14234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.824481 14041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:15.824798 14041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:15.825938 14041 hybrid_clock.cc:648] HybridClock initialized: now 1781967495825943 us; error 52 us; skew 500 ppm
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:15.827404 14041 init.cc:377] Logged in from keytab as oryx/127.7.46.188@KRBTEST.COM (short username oryx)
I20260620 14:58:15.827904 14041 webserver.cc:460] Webserver started at http://127.7.46.188:40875/ using document root <none> and password file <none>
I20260620 14:58:15.828101 14041 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:15.828158 14041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:15.829224 14041 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:58:15.830122 14243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.830345 14041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:58:15.830430 14041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
uuid: "d00d1e51f2c04f64b06a066b087582d0"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "5f9942000e4d3e32f05a27993d6cf738"
server_key_iv: "7e9ba4ca3fe63eaada8a7e40b5e01998"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.830549 14041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:15.862797 14041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:15.864915 14041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:15.870093 14041 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.188:33569
I20260620 14:58:15.870647 14041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
I20260620 14:58:15.870980 14295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.188:33569 every 8 connection(s)
I20260620 14:58:15.873348 14296 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:58:15.874342 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Bootstrap starting.
I20260620 14:58:15.881196 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:58:15.881489 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Bootstrap complete.
I20260620 14:58:15.881963 14296 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } }
I20260620 14:58:15.882103 14296 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d00d1e51f2c04f64b06a066b087582d0, State: Initialized, Role: LEARNER
I20260620 14:58:15.882217 14296 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } }
I20260620 14:58:15.882529 14302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:15.882685 14296 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:15.882687 14302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: LEARNER
W20260620 14:58:15.883713 14214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:15.884066 14214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:15.884142 14214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:58:15.885675 14313 catalog_manager.cc:1259] Loaded cluster ID: 7c740eac043144dc94b89e76b94c04f0
I20260620 14:58:15.885779 14313 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: loading cluster ID for follower catalog manager: success
I20260620 14:58:15.887491 14313 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: acquiring CA information for follower catalog manager: success
I20260620 14:58:15.888677 14313 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:58:15.890658 14214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:15.890726 14214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:15.890813 14214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:15.891865 13794 heartbeater.cc:503] Master 127.7.46.190:36619 was elected leader, sending a full tablet report...
I20260620 14:58:15.898793 14214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab
--principal=oryx/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:40267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=38931
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881,127.7.46.188:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:15.900029 14214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:15.901221 14214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:58:15.912034 14214 server_base.cc:1034] running on GCE node
W20260620 14:58:15.912006 14315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:15.912025 14316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:15.912218 14318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:15.912585 14214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:15.913223 14214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:15.914434 14214 hybrid_clock.cc:648] HybridClock initialized: now 1781967495914422 us; error 25 us; skew 500 ppm
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:15.917268 14214 init.cc:377] Logged in from keytab as oryx/127.7.46.129@KRBTEST.COM (short username oryx)
I20260620 14:58:15.918141 14214 webserver.cc:460] Webserver started at http://127.7.46.129:38931/ using document root <none> and password file <none>
I20260620 14:58:15.918702 14214 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:15.918771 14214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:15.922468 14214 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:58:15.924439 14325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.925625 14214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:58:15.925746 14214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "f270430196cb4484ac1e0eb57fd83b17"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "a70dad9fc0050b97a54bf40d0da226cd"
server_key_iv: "379b4d6914112b6ac83aa9ab33e42ba4"
server_key_version: "encryptionkey@0"
I20260620 14:58:15.926259 14214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:15.944594 14214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:15.948020 14214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:15.948221 14214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:15.948813 14214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:15.949687 14214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:15.949751 14214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.949818 14214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:15.949851 14214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:15.958720 14214 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:40267
I20260620 14:58:15.958789 14438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:40267 every 8 connection(s)
I20260620 14:58:15.959729 14214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:15.967600 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 14214
I20260620 14:58:15.967816 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 13665
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.188@KRBTEST.COM
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.189@KRBTEST.COM
Jun 20 14:58:15 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967495, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:15.976950 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.961658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:33029 (local address 127.7.46.189:37881)
0620 14:58:15.961809 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.961815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.962699 (+ 884us) server_negotiation.cc:407] Connection header received
0620 14:58:15.963808 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.963813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.963876 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.963957 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.965352 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.966158 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.967599 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.968023 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.968137 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.972794 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.972798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.972801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.972850 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.974507 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975103 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.975106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.975108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.975188 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975510 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.975513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.975515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.975719 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.975833 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.976663 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.976783 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:58:15.976953 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.961646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:57027 (local address 127.7.46.188:33569)
0620 14:58:15.961889 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.961894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.962698 (+ 804us) server_negotiation.cc:407] Connection header received
0620 14:58:15.964138 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.964141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.964183 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.964284 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.965358 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.966041 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.967610 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.968004 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.968079 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.972383 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.972387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.972390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.972417 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.974224 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975010 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.975014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.975018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.975107 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975510 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.975513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.975515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.976046 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.976139 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.976663 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.976784 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260620 14:58:15.977074 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.963203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:57289 (local address 127.7.46.190:36619)
0620 14:58:15.963327 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:15.963332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:15.963426 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:58:15.963889 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:15.963892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:15.963949 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:15.964020 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:15.965352 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.966156 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.967829 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.968151 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.968250 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.972438 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:15.972458 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:15.972464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:15.972540 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:15.974412 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975059 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.975065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.975070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.975160 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975533 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:15.975540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:15.975545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:15.975816 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:15.975894 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:15.976675 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 14:58:15.976813 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:15.978782 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.963070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.129:57289)
0620 14:58:15.963325 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.963340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.963593 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.964157 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.964166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.964754 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.965193 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.965209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.966296 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.966299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.967233 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.967242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.968624 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.968627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.968742 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.969477 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.969489 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.972269 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.974561 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.974567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.974572 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.974940 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.975275 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.975285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.975416 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.975977 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.975986 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.976369 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.977251 (+ 882us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.977593 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":167,"spinlock_wait_cycles":66176,"thread_start_us":74,"threads_started":1}
I20260620 14:58:15.978778 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.962089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:33569 (local address 127.7.46.129:57027)
0620 14:58:15.962550 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.962601 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.963576 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.964299 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.964302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.964711 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.965191 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.965204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.966180 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.966183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.967475 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.967485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.968269 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.968274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.968618 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.969565 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.969582 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.972204 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.974362 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.974368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.974386 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.974890 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.975219 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.975229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.975401 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.976129 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.976132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.976365 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.977101 (+ 736us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.977678 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":452,"spinlock_wait_cycles":26624,"thread_start_us":128,"threads_started":1}
I20260620 14:58:15.978778 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.962089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.7.46.129:33029)
0620 14:58:15.962550 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.962601 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:15.963583 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:15.964157 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:15.964166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:15.964745 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:15.965193 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.965209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.966258 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.966260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.967475 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:15.967485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:15.968274 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:15.968280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:15.968628 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:15.969429 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:15.969453 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:15.972602 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:15.974640 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.974645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.974648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.975003 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.975304 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:15.975306 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:15.975308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:15.975401 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:15.976203 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:15.976206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:15.976365 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 14:58:15.977727 (+ 1362us) client_negotiation.cc:240] Negotiation successful
0620 14:58:15.977840 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":360,"spinlock_wait_cycles":93056,"thread_start_us":130,"threads_started":1}
I20260620 14:58:15.979321 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:36791
--local_ip_for_outbound_sockets=127.7.46.130
--tserver_master_addrs=127.7.46.190:36619
--webserver_port=42323
--webserver_interface=127.7.46.130
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab
--principal=oryx/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881,127.7.46.188:33569 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:58:15.980218 14441 heartbeater.cc:347] Connected to a master server at 127.7.46.189:37881
I20260620 14:58:15.980471 14441 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:15.981060 14441 heartbeater.cc:511] Master 127.7.46.189:37881 requested a full tablet report, sending...
I20260620 14:58:15.981480 14440 heartbeater.cc:347] Connected to a master server at 127.7.46.188:33569
I20260620 14:58:15.981495 14439 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36619
I20260620 14:58:15.981566 14440 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:15.981606 14439 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:15.981715 14440 heartbeater.cc:511] Master 127.7.46.188:33569 requested a full tablet report, sending...
I20260620 14:58:15.981787 14439 heartbeater.cc:511] Master 127.7.46.190:36619 requested a full tablet report, sending...
I20260620 14:58:15.982717 13469 ts_manager.cc:194] Re-registered known tserver with Master: f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267)
I20260620 14:58:15.982745 14069 ts_manager.cc:194] Registered new tserver with Master: f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267)
I20260620 14:58:15.983644 13469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:57289
I20260620 14:58:15.983955 14260 ts_manager.cc:194] Registered new tserver with Master: f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267)
W20260620 14:58:16.091058 14447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:16.091290 14447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:16.091336 14447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:16.095463 14447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:16.095520 14447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:16.095579 14447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:16.100672 14447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab
--principal=oryx/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:36791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=42323
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881,127.7.46.188:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:16.101603 14447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:16.102567 14447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:58:16.106010 13928 heartbeater.cc:503] Master 127.7.46.190:36619 was elected leader, sending a full tablet report...
W20260620 14:58:16.112144 14455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:16.112150 14453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:16.112193 14452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:16.112393 14447 server_base.cc:1034] running on GCE node
I20260620 14:58:16.112867 14447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:16.113471 14447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:16.114656 14447 hybrid_clock.cc:648] HybridClock initialized: now 1781967496114647 us; error 30 us; skew 500 ppm
Jun 20 14:58:16 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:16.117420 14447 init.cc:377] Logged in from keytab as oryx/127.7.46.130@KRBTEST.COM (short username oryx)
I20260620 14:58:16.118346 14447 webserver.cc:460] Webserver started at http://127.7.46.130:42323/ using document root <none> and password file <none>
I20260620 14:58:16.118888 14447 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:16.119016 14447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:16.122983 14447 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:58:16.124986 14462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:16.126152 14447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:16.126286 14447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "eeb7a459db0242ec83f65b1d7e56b6a4"
format_stamp: "Formatted at 2026-06-20 14:58:14 on dist-test-slave-30t4"
server_key: "e194ec7ba9d1818ec51e618a99f49460"
server_key_iv: "702c939a4ff8fd214fcbccb8f714bf47"
server_key_version: "encryptionkey@0"
I20260620 14:58:16.126806 14447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:16.140779 14447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:16.143527 14447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:16.143998 14447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:16.144598 14447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:16.145546 14447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:16.145625 14447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:16.145692 14447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:16.145742 14447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:16.155035 14447 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:36791
I20260620 14:58:16.155056 14575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:36791 every 8 connection(s)
I20260620 14:58:16.156052 14447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:16.165961 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 14447
I20260620 14:58:16.166172 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 13799
Jun 20 14:58:16 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.189@KRBTEST.COM
Jun 20 14:58:16 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
Jun 20 14:58:16 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.188@KRBTEST.COM
I20260620 14:58:16.172871 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:42801
--local_ip_for_outbound_sockets=127.7.46.131
--tserver_master_addrs=127.7.46.190:36619
--webserver_port=42167
--webserver_interface=127.7.46.131
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab
--principal=oryx/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881,127.7.46.188:33569 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:58:16.177798 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.157786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:37727 (local address 127.7.46.188:33569)
0620 14:58:16.157938 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.157944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.158809 (+ 865us) server_negotiation.cc:407] Connection header received
0620 14:58:16.159972 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.159975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.160039 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.160123 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.161526 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.162311 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.163724 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.164070 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.164174 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.173282 (+ 9108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.173288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.173290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.173316 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.175235 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.175934 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.175938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.175940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.176033 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.176416 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.176419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.176421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.176655 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.176743 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.177511 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.177637 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:58:16.177798 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.157751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:48801 (local address 127.7.46.189:37881)
0620 14:58:16.157875 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.157879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.158804 (+ 925us) server_negotiation.cc:407] Connection header received
0620 14:58:16.160089 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.160095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.160151 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.160227 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.161526 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.162319 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.163460 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.163816 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.163948 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.173132 (+ 9184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.173137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.173140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.173172 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.175275 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.175944 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.175948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.175950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.176024 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.176416 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.176419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.176421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.176681 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.176758 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.177519 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.177635 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:58:16.177964 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.159309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:58025 (local address 127.7.46.190:36619)
0620 14:58:16.159448 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.159453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.159472 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:58:16.160271 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.160275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.160325 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.160425 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.161514 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.162049 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.163595 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.163977 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.164076 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.173163 (+ 9087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.173169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.173175 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.173209 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.175114 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.175893 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.175899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.175901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.175994 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.176434 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.176439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.176442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.176723 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.176814 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.177545 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.177693 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:16.179630 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.158201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:33569 (local address 127.7.46.130:37727)
0620 14:58:16.158657 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.158709 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.159728 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.160313 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.160322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.160858 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:16.161366 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.161379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.162766 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.162769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.163612 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.163619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.164298 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.164301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.164408 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.165207 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:16.165229 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:16.172905 (+ 7676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:16.175371 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.175374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.175376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.175815 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.176148 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.176151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.176153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.176304 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.176809 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:16.176818 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:16.177217 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.178567 (+ 1350us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.178684 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":195,"spinlock_wait_cycles":59392,"thread_start_us":125,"threads_started":1}
I20260620 14:58:16.179931 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.158201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.7.46.130:48801)
0620 14:58:16.158657 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.158709 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.159750 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.160313 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.160322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.160858 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:16.161366 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.161379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.162427 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.162430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.163332 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.163341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.164112 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.164116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.164325 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.165202 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:16.165236 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:16.172905 (+ 7669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:16.175377 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.175379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.175382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.175815 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.176148 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.176151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.176153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.176304 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.176826 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:16.176828 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:16.177217 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.178038 (+ 821us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.178450 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":275,"spinlock_wait_cycles":111872,"thread_start_us":149,"threads_started":1}
I20260620 14:58:16.179630 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.159177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.130:58025)
0620 14:58:16.159420 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.159430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.159787 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.160447 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.160450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.160844 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:16.161364 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.161377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.162192 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.162195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.163476 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.163488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.164112 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.164119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.164335 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.165207 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:16.165227 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:16.172998 (+ 7771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:16.175238 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.175248 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.175265 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.175732 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.176115 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.176121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.176123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.176316 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.176933 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:16.176938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:16.177214 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.178126 (+ 912us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.178546 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":241,"spinlock_wait_cycles":35968,"thread_start_us":73,"threads_started":1}
I20260620 14:58:16.181105 14578 heartbeater.cc:347] Connected to a master server at 127.7.46.189:37881
I20260620 14:58:16.181175 14576 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36619
I20260620 14:58:16.181339 14576 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:16.181367 14577 heartbeater.cc:347] Connected to a master server at 127.7.46.188:33569
I20260620 14:58:16.181433 14577 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:16.181869 14577 heartbeater.cc:511] Master 127.7.46.188:33569 requested a full tablet report, sending...
I20260620 14:58:16.181340 14578 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:16.182663 14578 heartbeater.cc:511] Master 127.7.46.189:37881 requested a full tablet report, sending...
I20260620 14:58:16.182940 14576 heartbeater.cc:511] Master 127.7.46.190:36619 requested a full tablet report, sending...
I20260620 14:58:16.183154 14260 ts_manager.cc:194] Registered new tserver with Master: eeb7a459db0242ec83f65b1d7e56b6a4 (127.7.46.130:36791)
I20260620 14:58:16.183468 14069 ts_manager.cc:194] Registered new tserver with Master: eeb7a459db0242ec83f65b1d7e56b6a4 (127.7.46.130:36791)
I20260620 14:58:16.184121 13469 ts_manager.cc:194] Re-registered known tserver with Master: eeb7a459db0242ec83f65b1d7e56b6a4 (127.7.46.130:36791)
I20260620 14:58:16.185010 13469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:58025
I20260620 14:58:16.232748 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.228975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.7.46.188:33569)
0620 14:58:16.229116 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.229122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.229138 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:16.229324 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.229329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.229389 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.229486 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:16.229775 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.230517 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.231745 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.232222 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.232347 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.232503 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.232548 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.232610 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:58:16.232748 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.228891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:33569 (local address 127.0.0.1:39078)
0620 14:58:16.229054 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.229072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.229181 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.229518 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.229521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.229563 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:16.229662 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.229668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.230627 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.230633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.231628 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.231638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.232351 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.232354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.232436 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.232459 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.232536 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.232610 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:58:16.233330 14266 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 14:58:16.233640 14302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6496092390a24fef98320c7d834845bc. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:16.233788 14302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: LEARNER
W20260620 14:58:16.289062 14583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:16.289297 14583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:16.289340 14583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:16.293385 14583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:16.293438 14583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:16.293495 14583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:16.298674 14583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab
--principal=oryx/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:42801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=42167
--tserver_master_addrs=127.7.46.190:36619,127.7.46.189:37881,127.7.46.188:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:16.299594 14583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:16.300639 14583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:16.310046 14589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:16.310031 14588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:16.310056 14591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:16.310253 14583 server_base.cc:1034] running on GCE node
I20260620 14:58:16.310782 14583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:16.311429 14583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:16.312618 14583 hybrid_clock.cc:648] HybridClock initialized: now 1781967496312598 us; error 33 us; skew 500 ppm
Jun 20 14:58:16 dist-test-slave-30t4 krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:16.315279 14583 init.cc:377] Logged in from keytab as oryx/127.7.46.131@KRBTEST.COM (short username oryx)
I20260620 14:58:16.316242 14583 webserver.cc:460] Webserver started at http://127.7.46.131:42167/ using document root <none> and password file <none>
I20260620 14:58:16.316744 14583 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:16.316812 14583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:16.320585 14583 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:16.322444 14598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:16.323613 14583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:16.323743 14583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "ad147cf6977d42b794ff577c738f277a"
format_stamp: "Formatted at 2026-06-20 14:58:15 on dist-test-slave-30t4"
server_key: "f47d53d134def574025474261d191af6"
server_key_iv: "89c6f9efc1c542fc01ed93c47f7ce0c5"
server_key_version: "encryptionkey@0"
I20260620 14:58:16.324286 14583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:16.342036 14583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:16.345384 14583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:16.345537 14583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:16.346055 14583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:16.346858 14583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:16.346903 14583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:16.346942 14583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:16.346963 14583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:16.355733 14583 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:42801
I20260620 14:58:16.355814 14711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:42801 every 8 connection(s)
I20260620 14:58:16.356832 14583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:16.361162 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 14583
Jun 20 14:58:16 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.188@KRBTEST.COM
Jun 20 14:58:16 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.189@KRBTEST.COM
Jun 20 14:58:16 dist-test-slave-30t4 krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967496, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:16.375888 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.363431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.7.46.190:36619)
0620 14:58:16.365185 (+ 1754us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.365190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.365205 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:58:16.365243 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.365246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.365291 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.365384 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.367038 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.367922 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.369033 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.369370 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.369476 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.371560 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.371564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.371567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.371591 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.373564 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.374360 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.374364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.374366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.374452 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.374752 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.374755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.374757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.374944 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.375067 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.375564 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.375674 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1681,"thread_start_us":94,"threads_started":1}
I20260620 14:58:16.375976 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.358578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:33795 (local address 127.7.46.189:37881)
0620 14:58:16.358758 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.358766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.359974 (+ 1208us) server_negotiation.cc:407] Connection header received
0620 14:58:16.360777 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.360780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.360830 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.360884 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.362867 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.363617 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.365509 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.365876 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.365972 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.369759 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.369764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.369769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.369798 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.371886 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.373149 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.373155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.373160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.373255 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.374543 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.374547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.374550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.374797 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.374894 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.375612 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.375733 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:58:16.378245 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.359056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.7.46.131:33795)
0620 14:58:16.359868 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.359879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.360583 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.361204 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.361213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.361840 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:16.362688 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.362709 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.363838 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.363841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.365384 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.365393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.366182 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.366186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.366423 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.367420 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:16.367435 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:16.369593 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:16.372359 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.372365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.372380 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.372709 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.373358 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.373360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.373362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.373526 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.374970 (+ 1444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:16.374976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:16.375313 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.376642 (+ 1329us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.377030 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":419,"spinlock_wait_cycles":22272,"thread_start_us":105,"threads_started":1}
I20260620 14:58:16.380211 14714 heartbeater.cc:347] Connected to a master server at 127.7.46.189:37881
I20260620 14:58:16.380471 14714 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:16.381071 14714 heartbeater.cc:511] Master 127.7.46.189:37881 requested a full tablet report, sending...
I20260620 14:58:16.382340 14069 ts_manager.cc:194] Registered new tserver with Master: ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801)
I20260620 14:58:16.379189 13469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:58:16.383183 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.369272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.7.46.189:37881)
0620 14:58:16.369783 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.369790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.370078 (+ 288us) server_negotiation.cc:407] Connection header received
0620 14:58:16.370118 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.370121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.370169 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.370249 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.371314 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.372149 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.373235 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.373568 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.373659 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.375238 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.375243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.375245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.375266 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.378483 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.378997 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.379002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.379004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.379087 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.382065 (+ 2978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.382068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.382070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.382277 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.382402 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.382885 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.383000 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":76,"threads_started":1}
W20260620 14:58:16.384065 13469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:16.386029 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.364134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.7.46.188:33569)
0620 14:58:16.364758 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.364769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.366733 (+ 1964us) server_negotiation.cc:407] Connection header received
0620 14:58:16.366774 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.366777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.366820 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.366917 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.367848 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.368686 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.369848 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.372163 (+ 2315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.372254 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.379390 (+ 7136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.379394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.379397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.379422 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.381376 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.383445 (+ 2069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.383451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.383452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.383556 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.384034 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.384039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.384042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.384282 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.384392 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.385712 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.385854 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":117,"threads_started":1}
I20260620 14:58:16.387192 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: NON_VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: true } } }
I20260620 14:58:16.387315 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:16.389282 14730 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: attempting to promote NON_VOTER d00d1e51f2c04f64b06a066b087582d0 to VOTER
I20260620 14:58:16.389672 14730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } }
I20260620 14:58:16.390782 14266 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 LEARNER]: Refusing update from remote peer 6496092390a24fef98320c7d834845bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:58:16.391127 14079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Refusing update from remote peer 6496092390a24fef98320c7d834845bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:58:16.391548 14028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:58:16.392195 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.358728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:54363 (local address 127.7.46.190:36619)
0620 14:58:16.358850 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.358855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.359692 (+ 837us) server_negotiation.cc:407] Connection header received
0620 14:58:16.360765 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.360769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.360828 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.360928 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.362861 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.363684 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.365549 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.365874 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.365946 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.388403 (+ 22457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.388407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.388410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.388435 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.390230 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.390759 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.390762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.390764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.390848 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.391193 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.391198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.391202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.391421 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.391520 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.391921 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.392037 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:58:16.392356 14028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:58:16.392731 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.359056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.131:54363)
0620 14:58:16.359517 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.359565 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.360565 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.361485 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.361490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.361873 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:16.362694 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.362709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.363844 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.363847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.365400 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.365410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.366138 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.366141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.366615 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.369804 (+ 3189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:16.369818 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:16.387858 (+ 18040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:16.390370 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.390373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.390375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.390636 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.390960 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.390966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.390968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.391068 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.391583 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:16.391586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:16.391683 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.392355 (+ 672us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.392519 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":360,"spinlock_wait_cycles":5248,"thread_start_us":123,"threads_started":1}
I20260620 14:58:16.393455 14712 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36619
I20260620 14:58:16.393553 14712 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:16.393743 14712 heartbeater.cc:511] Master 127.7.46.190:36619 requested a full tablet report, sending...
I20260620 14:58:16.393682 14731 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d00d1e51f2c04f64b06a066b087582d0 (127.7.46.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:16.394796 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.360102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:44391 (local address 127.7.46.188:33569)
0620 14:58:16.360225 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.360231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.360330 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:58:16.360776 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.360780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.360834 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.360884 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.362861 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.363677 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.365540 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.365872 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.365946 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.390785 (+ 24839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.390789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.390794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.390823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.392566 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.393417 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.393421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.393422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.393517 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.393845 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.393849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.393850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.394052 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.394123 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.394541 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.394646 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:58:16.394817 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.359949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:33569 (local address 127.7.46.131:44391)
0620 14:58:16.360219 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.360233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.360582 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.361204 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.361213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.361885 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:16.362694 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.362709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.363844 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.363847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.365400 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.365410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.366138 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.366141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.366419 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.367169 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:16.367194 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:16.388243 (+ 21049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:16.392867 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.392870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.392872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.393270 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.393642 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:16.393645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:16.393647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:16.393740 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:16.394196 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:16.394200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:16.394328 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.394451 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.394635 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":339,"spinlock_wait_cycles":6400,"thread_start_us":95,"threads_started":1}
I20260620 14:58:16.395088 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:16.395208 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:16.395274 14713 heartbeater.cc:347] Connected to a master server at 127.7.46.188:33569
I20260620 14:58:16.395349 14713 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:16.395334 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:16.395411 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:16.395553 14713 heartbeater.cc:511] Master 127.7.46.188:33569 requested a full tablet report, sending...
I20260620 14:58:16.396144 13468 ts_manager.cc:194] Re-registered known tserver with Master: ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801)
I20260620 14:58:16.396267 14260 ts_manager.cc:194] Registered new tserver with Master: ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801)
I20260620 14:58:16.396826 13468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:54363
I20260620 14:58:16.397363 14079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d00d1e51f2c04f64b06a066b087582d0 (127.7.46.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:16.398767 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:16.399086 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:58:16.399376 14730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:16.399478 14730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:16.399850 14266 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d00d1e51f2c04f64b06a066b087582d0 (127.7.46.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:16.401242 14302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6496092390a24fef98320c7d834845bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:16.401362 14302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:58:16.411020 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.405437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:42801 (local address 127.0.0.1:56756)
0620 14:58:16.405935 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.405950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.406060 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.406963 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.406967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.406984 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:16.407098 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.407105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.408418 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.408422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.409406 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.409415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.410608 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.410612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.410698 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.410724 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.410806 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.410877 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":20,"spinlock_wait_cycles":9600}
I20260620 14:58:16.411063 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.405543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.7.46.131:42801)
0620 14:58:16.406327 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.406343 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.406369 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:58:16.406443 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.406449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.406663 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.406828 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:16.407215 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.408303 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.409533 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.410318 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.410414 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.410547 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.410851 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.410938 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":103,"threads_started":1}
I20260620 14:58:16.411973 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.406260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:40267 (local address 127.0.0.1:60078)
0620 14:58:16.407160 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.407176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.407261 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.407839 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.407842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.407864 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:16.407961 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.407967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.409529 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.409533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.410299 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.410308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.411565 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.411570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.411654 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.411676 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.411744 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.411833 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":17,"thread_start_us":84,"threads_started":1}
I20260620 14:58:16.412242 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.405720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:36791 (local address 127.0.0.1:48984)
0620 14:58:16.405946 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.405962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.406043 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.407926 (+ 1883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.407929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.407945 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:16.408022 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.408027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.409348 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.409352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.410527 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.410537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.411868 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.411874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.411975 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.411999 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.412049 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.412120 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":43,"thread_start_us":83,"threads_started":1}
I20260620 14:58:16.412345 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.406413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.7.46.129:40267)
0620 14:58:16.406723 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.406737 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.407274 (+ 537us) server_negotiation.cc:407] Connection header received
0620 14:58:16.407365 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.407370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.407567 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.407717 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:16.408104 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.409415 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.410426 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.411213 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.411317 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.411447 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.412123 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.412215 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":127,"threads_started":1}
I20260620 14:58:16.413007 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.405882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.7.46.130:36791)
0620 14:58:16.407467 (+ 1585us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.407476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.407492 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:16.407558 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.407562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.407773 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.407955 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:16.408137 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.409230 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.410648 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.411708 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.412696 (+ 988us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.412798 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.412836 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.412900 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":110,"threads_started":1}
I20260620 14:58:16.413880 14646 tablet_service.cc:1467] Processing CreateTablet for tablet 066b864b175e48298d84c69e21852d3a (DEFAULT_TABLE table=test-table [id=3d511d59bdd149a0b63331329f69c0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:16.414811 14646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 066b864b175e48298d84c69e21852d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:16.415246 14510 tablet_service.cc:1467] Processing CreateTablet for tablet 066b864b175e48298d84c69e21852d3a (DEFAULT_TABLE table=test-table [id=3d511d59bdd149a0b63331329f69c0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:16.416311 14510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 066b864b175e48298d84c69e21852d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:16.421388 14373 tablet_service.cc:1467] Processing CreateTablet for tablet 066b864b175e48298d84c69e21852d3a (DEFAULT_TABLE table=test-table [id=3d511d59bdd149a0b63331329f69c0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:16.421939 14745 tablet_bootstrap.cc:492] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: Bootstrap starting.
I20260620 14:58:16.422384 14373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 066b864b175e48298d84c69e21852d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:16.423807 14746 tablet_bootstrap.cc:492] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: Bootstrap starting.
I20260620 14:58:16.424376 14745 tablet_bootstrap.cc:654] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:16.425247 14745 log.cc:824] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:16.425896 14746 tablet_bootstrap.cc:654] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:16.426635 14746 log.cc:824] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:16.428567 14745 tablet_bootstrap.cc:492] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: No bootstrap required, opened a new log
I20260620 14:58:16.428568 14746 tablet_bootstrap.cc:492] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: No bootstrap required, opened a new log
I20260620 14:58:16.428759 14746 ts_tablet_manager.cc:1397] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:58:16.428759 14745 ts_tablet_manager.cc:1397] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:58:16.429123 14749 tablet_bootstrap.cc:492] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Bootstrap starting.
I20260620 14:58:16.431428 14749 tablet_bootstrap.cc:654] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:16.432229 14749 log.cc:824] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:16.433804 14746 raft_consensus.cc:348] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.433804 14745 raft_consensus.cc:348] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.434062 14745 raft_consensus.cc:374] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:16.434062 14746 raft_consensus.cc:374] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:16.434121 14746 raft_consensus.cc:729] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb7a459db0242ec83f65b1d7e56b6a4, State: Initialized, Role: FOLLOWER
I20260620 14:58:16.434121 14745 raft_consensus.cc:729] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad147cf6977d42b794ff577c738f277a, State: Initialized, Role: FOLLOWER
I20260620 14:58:16.434221 14749 tablet_bootstrap.cc:492] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: No bootstrap required, opened a new log
I20260620 14:58:16.434398 14749 ts_tablet_manager.cc:1397] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:58:16.434572 14746 consensus_queue.cc:260] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.434572 14745 consensus_queue.cc:260] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.435441 14576 heartbeater.cc:503] Master 127.7.46.190:36619 was elected leader, sending a full tablet report...
I20260620 14:58:16.435549 14746 ts_tablet_manager.cc:1428] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:58:16.435546 14745 ts_tablet_manager.cc:1428] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 14:58:16.435993 14712 heartbeater.cc:503] Master 127.7.46.190:36619 was elected leader, sending a full tablet report...
I20260620 14:58:16.438748 14749 raft_consensus.cc:348] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.439004 14749 raft_consensus.cc:374] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:16.439069 14749 raft_consensus.cc:729] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f270430196cb4484ac1e0eb57fd83b17, State: Initialized, Role: FOLLOWER
I20260620 14:58:16.439535 14749 consensus_queue.cc:260] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.440124 14439 heartbeater.cc:503] Master 127.7.46.190:36619 was elected leader, sending a full tablet report...
I20260620 14:58:16.440483 14749 ts_tablet_manager.cc:1428] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260620 14:58:16.444947 14752 raft_consensus.cc:482] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:16.445117 14752 raft_consensus.cc:504] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.446074 14752 leader_election.cc:288] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801), f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267)
I20260620 14:58:16.449738 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.446283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:44197 (local address 127.7.46.131:42801)
0620 14:58:16.446461 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.446467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.446483 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:16.446534 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.446536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.446578 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.446656 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:16.446910 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.447572 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.448779 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.449169 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.449298 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.449346 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.449540 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.449621 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 14:58:16.449743 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.446224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:42801 (local address 127.7.46.130:44197)
0620 14:58:16.446328 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.446344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.446422 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.446678 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.446681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.446696 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:16.446814 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.446823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.447675 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.447680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.448662 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.448671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.449286 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.449289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.449391 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.449412 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.449508 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.449603 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":33}
I20260620 14:58:16.449767 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.446268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:40267 (local address 127.7.46.130:36615)
0620 14:58:16.446405 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.446419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:16.446496 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:16.447099 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:16.447102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:16.447120 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:16.447220 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.447226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.447917 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.447921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.448694 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:16.448703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.449348 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.449352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:16.449453 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.449498 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:58:16.449587 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:58:16.449651 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":15}
I20260620 14:58:16.449779 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.446800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:36615 (local address 127.7.46.129:40267)
0620 14:58:16.446916 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.446919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.446929 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:58:16.446962 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.446964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.446995 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.447062 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:16.447320 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.447825 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.448821 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.449223 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.449353 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.449398 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.449613 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.449685 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:58:16.450373 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066b864b175e48298d84c69e21852d3a" candidate_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad147cf6977d42b794ff577c738f277a" is_pre_election: true
I20260620 14:58:16.450665 14666 raft_consensus.cc:2454] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb7a459db0242ec83f65b1d7e56b6a4 in term 0.
I20260620 14:58:16.450835 14393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066b864b175e48298d84c69e21852d3a" candidate_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f270430196cb4484ac1e0eb57fd83b17" is_pre_election: true
I20260620 14:58:16.451121 14393 raft_consensus.cc:2454] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb7a459db0242ec83f65b1d7e56b6a4 in term 0.
I20260620 14:58:16.451160 14466 leader_election.cc:302] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad147cf6977d42b794ff577c738f277a, eeb7a459db0242ec83f65b1d7e56b6a4; no voters:
I20260620 14:58:16.451428 14752 raft_consensus.cc:2781] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:16.451516 14752 raft_consensus.cc:482] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:16.451576 14752 raft_consensus.cc:3037] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:16.452739 14752 raft_consensus.cc:504] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.453145 14752 leader_election.cc:288] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [CANDIDATE]: Term 1 election: Requested vote from peers ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801), f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267)
I20260620 14:58:16.453444 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066b864b175e48298d84c69e21852d3a" candidate_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad147cf6977d42b794ff577c738f277a"
I20260620 14:58:16.453476 14393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066b864b175e48298d84c69e21852d3a" candidate_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f270430196cb4484ac1e0eb57fd83b17"
I20260620 14:58:16.453575 14666 raft_consensus.cc:3037] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:16.453575 14393 raft_consensus.cc:3037] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:16.454823 14666 raft_consensus.cc:2454] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb7a459db0242ec83f65b1d7e56b6a4 in term 1.
I20260620 14:58:16.454823 14393 raft_consensus.cc:2454] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb7a459db0242ec83f65b1d7e56b6a4 in term 1.
I20260620 14:58:16.455093 14466 leader_election.cc:302] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad147cf6977d42b794ff577c738f277a, eeb7a459db0242ec83f65b1d7e56b6a4; no voters:
I20260620 14:58:16.455277 14752 raft_consensus.cc:2781] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:16.455516 14752 raft_consensus.cc:686] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 LEADER]: Becoming Leader. State: Replica: eeb7a459db0242ec83f65b1d7e56b6a4, State: Running, Role: LEADER
I20260620 14:58:16.455876 14752 consensus_queue.cc:237] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } }
I20260620 14:58:16.459525 13469 catalog_manager.cc:5495] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb7a459db0242ec83f65b1d7e56b6a4 (127.7.46.130). New cstate: current_term: 1 leader_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } health_report { overall_health: UNKNOWN } } }
W20260620 14:58:16.461823 14442 tablet.cc:2369] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:16.473433 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.469892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.7.46.130:36791)
0620 14:58:16.470038 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.470042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.470057 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:58:16.470226 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.470229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.470289 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.470395 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:16.470643 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.471146 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.472051 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.472281 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.472392 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.472686 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:16.473056 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:16.473184 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.473246 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.473299 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:58:16.478986 14393 raft_consensus.cc:1261] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 1 FOLLOWER]: Refusing update from remote peer eeb7a459db0242ec83f65b1d7e56b6a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:16.478986 14666 raft_consensus.cc:1261] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 1 FOLLOWER]: Refusing update from remote peer eeb7a459db0242ec83f65b1d7e56b6a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:16.479723 14754 consensus_queue.cc:1035] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:16.479924 14752 consensus_queue.cc:1035] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:16.489599 14758 mvcc.cc:205] Tried to move back new op lower bound from 7298938865571037184 to 7298938865485717504. Current Snapshot: MvccSnapshot[applied={T|T < 7298938865571037184}]
I20260620 14:58:16.499464 13468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:16.499639 13468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:16.503602 13468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6496092390a24fef98320c7d834845bc: Sending DeleteTablet for 3 replicas of tablet 066b864b175e48298d84c69e21852d3a
I20260620 14:58:16.504302 14646 tablet_service.cc:1514] Processing DeleteTablet for tablet 066b864b175e48298d84c69e21852d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:16 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:56756
I20260620 14:58:16.504395 14510 tablet_service.cc:1514] Processing DeleteTablet for tablet 066b864b175e48298d84c69e21852d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:16 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:48984
I20260620 14:58:16.504899 14767 tablet_replica.cc:330] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: stopping tablet replica
I20260620 14:58:16.505226 14373 tablet_service.cc:1514] Processing DeleteTablet for tablet 066b864b175e48298d84c69e21852d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:16 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:60078
I20260620 14:58:16.505681 14767 raft_consensus.cc:2229] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:16.505998 14767 raft_consensus.cc:2258] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:16.506480 14769 tablet_replica.cc:330] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: stopping tablet replica
I20260620 14:58:16.506798 14769 raft_consensus.cc:2229] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:16.507087 14769 raft_consensus.cc:2258] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:16.508529 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 13437
I20260620 14:58:16.509589 14767 ts_tablet_manager.cc:1905] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:16.512287 14767 ts_tablet_manager.cc:1918] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:16.512387 14767 log.cc:1196] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/066b864b175e48298d84c69e21852d3a
I20260620 14:58:16.512720 14767 ts_tablet_manager.cc:1939] T 066b864b175e48298d84c69e21852d3a P ad147cf6977d42b794ff577c738f277a: Deleting consensus metadata
I20260620 14:58:16.514554 14769 ts_tablet_manager.cc:1905] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:16.516192 14768 tablet_replica.cc:330] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: stopping tablet replica
I20260620 14:58:16.516489 14768 raft_consensus.cc:2229] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:16.516904 14768 raft_consensus.cc:2258] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:16.517068 14769 ts_tablet_manager.cc:1918] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:16.517150 14769 log.cc:1196] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/066b864b175e48298d84c69e21852d3a
I20260620 14:58:16.517467 14769 ts_tablet_manager.cc:1939] T 066b864b175e48298d84c69e21852d3a P f270430196cb4484ac1e0eb57fd83b17: Deleting consensus metadata
W20260620 14:58:16.517830 14601 connection.cc:535] server connection from 127.0.0.1:56756 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:58:16.518378 14768 ts_tablet_manager.cc:1905] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:16.520216 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.507376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.7.46.189:37881)
0620 14:58:16.507525 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.507532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.508317 (+ 785us) server_negotiation.cc:407] Connection header received
0620 14:58:16.508494 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.508497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.508571 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.508631 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.510075 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.510980 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.512459 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.512923 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.513010 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.515297 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.515304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.515310 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.515343 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.517383 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.518698 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.518704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.518708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.518815 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.519162 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.519167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.519171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.519412 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.519533 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.519920 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.520060 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 14:58:16.520431 14329 connection.cc:441] server connection from 127.0.0.1:60078 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 1) could send its response
I20260620 14:58:16.520957 14768 ts_tablet_manager.cc:1918] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:16.521070 14768 log.cc:1196] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/066b864b175e48298d84c69e21852d3a
I20260620 14:58:16.521430 14768 ts_tablet_manager.cc:1939] T 066b864b175e48298d84c69e21852d3a P eeb7a459db0242ec83f65b1d7e56b6a4: Deleting consensus metadata
W20260620 14:58:16.522367 14463 connection.cc:441] server connection from 127.0.0.1:48984 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48984 (request call id 1) could send its response
I20260620 14:58:16.530242 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.507175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.7.46.188:33569)
0620 14:58:16.507422 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.507426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.520560 (+ 13134us) server_negotiation.cc:407] Connection header received
0620 14:58:16.520829 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.520832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.520888 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.520948 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.521927 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.522747 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.523931 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.524263 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.524333 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.526004 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.526008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.526011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.526035 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.528216 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.528798 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.528801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.528803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.528883 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.529167 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.529171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.529173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.529365 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.529527 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.529911 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.530081 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":165}
I20260620 14:58:16.532574 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.521255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.7.46.188:33569)
0620 14:58:16.521421 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.521425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.522393 (+ 968us) server_negotiation.cc:407] Connection header received
0620 14:58:16.522538 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.522541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.522583 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.522626 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.524147 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.524912 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.526236 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.526576 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.526636 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.528548 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.528552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.528554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.528579 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.530735 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.531206 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.531210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.531211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.531293 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.531604 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.531610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.531615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.531872 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.531967 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.532286 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.532392 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":78}
I20260620 14:58:16.532927 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.521916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.7.46.189:37881)
0620 14:58:16.522012 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:16.522016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:16.522887 (+ 871us) server_negotiation.cc:407] Connection header received
0620 14:58:16.523058 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:16.523061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:16.523117 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:16.523175 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:16.524189 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.524927 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.526192 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:16.526521 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:16.526594 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:16.528554 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:16.528557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:16.528560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:16.528580 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:16.530640 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.531635 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.531638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.531640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.531712 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:16.532043 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:16.532047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:16.532049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:16.532249 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:16.532364 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:16.532681 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:58:16.532789 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:58:17.492832 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:17.492061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.129:58077)
0620 14:58:17.492608 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:17.492709 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":76,"threads_started":1}
W20260620 14:58:17.493280 14439 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:36619 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
I20260620 14:58:17.493317 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:17.492061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.131:50329)
0620 14:58:17.493129 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:17.493220 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":79,"threads_started":1}
W20260620 14:58:17.493687 14712 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:36619 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
I20260620 14:58:17.494066 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:17.492593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.130:57167)
0620 14:58:17.493867 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:17.493954 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1191,"thread_start_us":84,"threads_started":1}
W20260620 14:58:17.494341 14576 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:36619 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
I20260620 14:58:18.004719 14781 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6496092390a24fef98320c7d834845bc)
I20260620 14:58:18.004906 14781 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } }
I20260620 14:58:18.005908 14781 leader_election.cc:288] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6496092390a24fef98320c7d834845bc (127.7.46.190:36619), 65ffb39156b04715b3ba9b10e07387e9 (127.7.46.189:37881)
I20260620 14:58:18.006394 14784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6496092390a24fef98320c7d834845bc)
I20260620 14:58:18.006553 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.006070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:60426)
0620 14:58:18.006358 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.006441 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":59,"threads_started":1}
I20260620 14:58:18.006594 14784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } }
W20260620 14:58:18.006729 14246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
I20260620 14:58:18.007597 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.007357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:60428)
0620 14:58:18.007474 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.007533 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:58:18.007644 14784 leader_election.cc:288] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6496092390a24fef98320c7d834845bc (127.7.46.190:36619), d00d1e51f2c04f64b06a066b087582d0 (127.7.46.188:33569)
W20260620 14:58:18.007683 14246 leader_election.cc:334] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6496092390a24fef98320c7d834845bc (127.7.46.190:36619): Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
I20260620 14:58:18.008116 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.007704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:60438)
0620 14:58:18.007948 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.008025 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
W20260620 14:58:18.008294 14055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
I20260620 14:58:18.009126 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.008890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:60446)
0620 14:58:18.009004 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.009059 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:58:18.009280 14055 leader_election.cc:334] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6496092390a24fef98320c7d834845bc (127.7.46.190:36619): Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
I20260620 14:58:18.010346 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.006139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37881 (local address 127.0.0.1:35538)
0620 14:58:18.006423 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.006438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:18.006544 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:18.006926 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:18.006929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:18.006950 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:18.007064 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.007071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.008094 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.008097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.009233 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.009243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.009935 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.009938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.010041 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.010062 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:58:18.010156 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:58:18.010223 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":57,"threads_started":1}
I20260620 14:58:18.010349 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.006224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.7.46.189:37881)
0620 14:58:18.006559 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:18.006567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:18.006590 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:58:18.006737 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:18.006743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:18.006797 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:18.006895 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:18.007191 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.007948 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.009363 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.009817 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.009934 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.010061 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:18.010169 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:58:18.010243 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":178,"thread_start_us":61,"threads_started":1}
I20260620 14:58:18.010778 14079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d00d1e51f2c04f64b06a066b087582d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65ffb39156b04715b3ba9b10e07387e9" is_pre_election: true
I20260620 14:58:18.010957 14079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00d1e51f2c04f64b06a066b087582d0 in term 1.
I20260620 14:58:18.011289 14244 leader_election.cc:302] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65ffb39156b04715b3ba9b10e07387e9, d00d1e51f2c04f64b06a066b087582d0; no voters: 6496092390a24fef98320c7d834845bc
I20260620 14:58:18.011505 14781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:58:18.011577 14781 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6496092390a24fef98320c7d834845bc)
I20260620 14:58:18.011632 14781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:58:18.012253 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.008064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:33569 (local address 127.0.0.1:39114)
0620 14:58:18.008236 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.008249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:18.008364 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:18.008762 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:18.008765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:18.008784 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:18.008862 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.008867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.009954 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.009959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.011004 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.011014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.011846 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.011850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.011941 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.011962 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:58:18.012065 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:58:18.012133 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":22,"thread_start_us":60,"threads_started":1}
I20260620 14:58:18.012262 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.008166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.7.46.188:33569)
0620 14:58:18.008532 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:18.008541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:18.008555 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:18.008595 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:18.008598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:18.008644 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:18.008736 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:18.009284 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.009837 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.011125 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.011699 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.011835 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.011884 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:18.012079 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:58:18.012147 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":92,"threads_started":1}
I20260620 14:58:18.012703 14266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65ffb39156b04715b3ba9b10e07387e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d00d1e51f2c04f64b06a066b087582d0" is_pre_election: true
I20260620 14:58:18.012847 14781 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } }
I20260620 14:58:18.013105 14266 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65ffb39156b04715b3ba9b10e07387e9 in current term 2: Already voted for candidate d00d1e51f2c04f64b06a066b087582d0 in this term.
I20260620 14:58:18.013233 14781 leader_election.cc:288] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [CANDIDATE]: Term 2 election: Requested vote from peers 6496092390a24fef98320c7d834845bc (127.7.46.190:36619), 65ffb39156b04715b3ba9b10e07387e9 (127.7.46.189:37881)
I20260620 14:58:18.013520 14055 leader_election.cc:302] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65ffb39156b04715b3ba9b10e07387e9; no voters: 6496092390a24fef98320c7d834845bc, d00d1e51f2c04f64b06a066b087582d0
I20260620 14:58:18.013526 14079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d00d1e51f2c04f64b06a066b087582d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65ffb39156b04715b3ba9b10e07387e9"
I20260620 14:58:18.013625 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.013347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:60448)
0620 14:58:18.013514 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.013569 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 14:58:18.013680 14079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:58:18.014554 14079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00d1e51f2c04f64b06a066b087582d0 in term 2.
I20260620 14:58:18.014667 14784 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:58:18.014851 14244 leader_election.cc:302] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65ffb39156b04715b3ba9b10e07387e9, d00d1e51f2c04f64b06a066b087582d0; no voters:
I20260620 14:58:18.015031 14781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:58:18.015205 14781 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [term 2 LEADER]: Becoming Leader. State: Replica: d00d1e51f2c04f64b06a066b087582d0, State: Running, Role: LEADER
I20260620 14:58:18.015235 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.015005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:60460)
0620 14:58:18.015121 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.015177 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:58:18.015321 14246 leader_election.cc:334] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6496092390a24fef98320c7d834845bc (127.7.46.190:36619): Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
I20260620 14:58:18.015458 14781 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } }
I20260620 14:58:18.016415 14791 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00d1e51f2c04f64b06a066b087582d0. Latest consensus state: current_term: 2 leader_uuid: "d00d1e51f2c04f64b06a066b087582d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:18.016503 14791 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:18.016785 14793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:18.018903 14793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:18.019341 14793 catalog_manager.cc:1259] Loaded cluster ID: 7c740eac043144dc94b89e76b94c04f0
I20260620 14:58:18.019384 14793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:18.019956 14793 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:18.020350 14793 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Loaded TSK: 0
I20260620 14:58:18.020972 14793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:18.051544 14260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:18.054597 14260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:18.057456 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.057227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:60466)
0620 14:58:18.057342 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.057394 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:58:18.057502 14079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [term 2 FOLLOWER]: Refusing update from remote peer d00d1e51f2c04f64b06a066b087582d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 14:58:18.058045 14781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 14:58:18.058179 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.057960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.0.0.1:60474)
0620 14:58:18.058073 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.058122 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:58:18.058290 14246 consensus_peers.cc:487] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 -> Peer 6496092390a24fef98320c7d834845bc (127.7.46.190:36619): Couldn't send request to peer 6496092390a24fef98320c7d834845bc. Status: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:58:18.059499 14784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00d1e51f2c04f64b06a066b087582d0. Latest consensus state: current_term: 2 leader_uuid: "d00d1e51f2c04f64b06a066b087582d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:18.059615 14784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:58:18.060600 14791 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d00d1e51f2c04f64b06a066b087582d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:18.060726 14791 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:18.060860 14795 mvcc.cc:205] Tried to move back new op lower bound from 7298938872039325696 to 7298938871873560576. Current Snapshot: MvccSnapshot[applied={T|T < 7298938872039325696}]
I20260620 14:58:18.061102 14784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d00d1e51f2c04f64b06a066b087582d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6496092390a24fef98320c7d834845bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36619 } } peers { permanent_uuid: "65ffb39156b04715b3ba9b10e07387e9" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37881 } attrs { promote: false } } peers { permanent_uuid: "d00d1e51f2c04f64b06a066b087582d0" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 33569 } attrs { promote: false } } }
I20260620 14:58:18.061223 14784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ffb39156b04715b3ba9b10e07387e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:58:18.070709 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.066271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:40267 (local address 127.0.0.1:60088)
0620 14:58:18.066420 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.066434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:18.066508 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:18.066893 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:18.066899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:18.066920 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:18.067017 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.067023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.068179 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.068182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.069446 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.069455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.070307 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.070311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.070411 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.070444 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:58:18.070494 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:58:18.070570 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":30}
I20260620 14:58:18.070842 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.066578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:36791 (local address 127.0.0.1:49002)
0620 14:58:18.066660 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.066673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:18.066740 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:18.067357 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:18.067360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:18.067377 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:18.067456 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.067462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.068434 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.068437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.069567 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.069576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.070434 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.070437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.070526 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.070550 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:58:18.070593 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:58:18.070655 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":2}
I20260620 14:58:18.071086 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.066832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.7.46.130:36791)
0620 14:58:18.067101 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:18.067105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:18.067117 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:58:18.067168 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:18.067173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:18.067226 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:18.067343 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:18.067586 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.068345 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.069680 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.070314 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.070776 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.070846 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:18.070883 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:58:18.070956 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260620 14:58:18.071419 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.067224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:42801 (local address 127.0.0.1:56762)
0620 14:58:18.067947 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.067959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:18.068046 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:18.068290 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:18.068293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:18.068308 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:18.068447 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.068453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.069154 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.069157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.070082 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:18.070091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.071034 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.071037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:18.071120 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.071146 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:58:18.071218 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:58:18.071281 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260620 14:58:18.071625 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.066398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.7.46.129:40267)
0620 14:58:18.066662 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:18.066668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:18.066684 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:18.066736 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:18.066741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:18.066791 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:18.066885 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:18.067244 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.068056 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.069585 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.070188 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.071308 (+ 1120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.071395 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:18.071430 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:58:18.071493 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260620 14:58:18.071578 14510 tablet_service.cc:1467] Processing CreateTablet for tablet c9965d8b0c1d41a880d8ca036933cc32 (DEFAULT_TABLE table=test-table [id=afc1ab1a21354304a4e4336e68d8b5e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:18.071862 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.067575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.7.46.131:42801)
0620 14:58:18.067831 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:18.067835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:18.068046 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:58:18.068128 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:18.068133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:18.068189 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:18.068271 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:18.068541 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.069038 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.070218 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.070836 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.070929 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.070982 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:18.071587 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 14:58:18.071663 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260620 14:58:18.071938 14510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9965d8b0c1d41a880d8ca036933cc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:18.072120 14373 tablet_service.cc:1467] Processing CreateTablet for tablet c9965d8b0c1d41a880d8ca036933cc32 (DEFAULT_TABLE table=test-table [id=afc1ab1a21354304a4e4336e68d8b5e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:18.072347 14373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9965d8b0c1d41a880d8ca036933cc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:18.072338 14646 tablet_service.cc:1467] Processing CreateTablet for tablet c9965d8b0c1d41a880d8ca036933cc32 (DEFAULT_TABLE table=test-table [id=afc1ab1a21354304a4e4336e68d8b5e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:18.072562 14646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9965d8b0c1d41a880d8ca036933cc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:18.074407 14804 tablet_bootstrap.cc:492] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17: Bootstrap starting.
I20260620 14:58:18.074723 14805 tablet_bootstrap.cc:492] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: Bootstrap starting.
I20260620 14:58:18.074962 14806 tablet_bootstrap.cc:492] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: Bootstrap starting.
I20260620 14:58:18.075351 14804 tablet_bootstrap.cc:654] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:18.075582 14805 tablet_bootstrap.cc:654] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:18.075886 14806 tablet_bootstrap.cc:654] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:18.076640 14804 tablet_bootstrap.cc:492] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17: No bootstrap required, opened a new log
I20260620 14:58:18.076743 14804 ts_tablet_manager.cc:1397] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:58:18.076828 14805 tablet_bootstrap.cc:492] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: No bootstrap required, opened a new log
I20260620 14:58:18.076962 14805 ts_tablet_manager.cc:1397] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:18.077111 14806 tablet_bootstrap.cc:492] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: No bootstrap required, opened a new log
I20260620 14:58:18.077203 14806 ts_tablet_manager.cc:1397] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:58:18.077152 14804 raft_consensus.cc:348] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.077303 14804 raft_consensus.cc:374] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:18.077354 14804 raft_consensus.cc:729] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f270430196cb4484ac1e0eb57fd83b17, State: Initialized, Role: FOLLOWER
I20260620 14:58:18.077479 14804 consensus_queue.cc:260] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.077682 14806 raft_consensus.cc:348] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.077801 14806 raft_consensus.cc:374] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:18.077832 14804 ts_tablet_manager.cc:1428] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:58:18.077855 14806 raft_consensus.cc:729] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad147cf6977d42b794ff577c738f277a, State: Initialized, Role: FOLLOWER
I20260620 14:58:18.077739 14805 raft_consensus.cc:348] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.078023 14805 raft_consensus.cc:374] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:18.078017 14806 consensus_queue.cc:260] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.078096 14805 raft_consensus.cc:729] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb7a459db0242ec83f65b1d7e56b6a4, State: Initialized, Role: FOLLOWER
I20260620 14:58:18.078358 14806 ts_tablet_manager.cc:1428] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 14:58:18.078307 14805 consensus_queue.cc:260] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.078749 14805 ts_tablet_manager.cc:1428] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:18.080263 14440 heartbeater.cc:503] Master 127.7.46.188:33569 was elected leader, sending a full tablet report...
I20260620 14:58:18.080811 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.078743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.131:46961)
0620 14:58:18.080653 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.080711 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1843,"thread_start_us":100,"threads_started":1}
I20260620 14:58:18.081594 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.081387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.131:34619)
0620 14:58:18.081489 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.081537 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:58:18.081643 14713 heartbeater.cc:503] Master 127.7.46.188:33569 was elected leader, sending a full tablet report...
I20260620 14:58:18.082377 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.079220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.130:56557)
0620 14:58:18.082218 (+ 2998us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.082276 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1995,"thread_start_us":92,"threads_started":1}
I20260620 14:58:18.082768 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.078212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.129:58303)
0620 14:58:18.082611 (+ 4399us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.082666 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4323,"thread_start_us":84,"threads_started":1}
I20260620 14:58:18.083273 14577 heartbeater.cc:503] Master 127.7.46.188:33569 was elected leader, sending a full tablet report...
I20260620 14:58:18.083711 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.083202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.129:34667)
0620 14:58:18.083601 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.083652 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326}
I20260620 14:58:18.083724 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.083079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.130:57821)
0620 14:58:18.083193 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.083660 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:58:18.109280 14715 tablet.cc:2369] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:18.159085 14579 tablet.cc:2369] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:18.175169 14813 raft_consensus.cc:482] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:18.175348 14813 raft_consensus.cc:504] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.175789 14813 leader_election.cc:288] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267), ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801)
I20260620 14:58:18.176262 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9965d8b0c1d41a880d8ca036933cc32" candidate_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad147cf6977d42b794ff577c738f277a" is_pre_election: true
I20260620 14:58:18.176321 14393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9965d8b0c1d41a880d8ca036933cc32" candidate_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f270430196cb4484ac1e0eb57fd83b17" is_pre_election: true
I20260620 14:58:18.176424 14666 raft_consensus.cc:2454] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb7a459db0242ec83f65b1d7e56b6a4 in term 0.
I20260620 14:58:18.176488 14393 raft_consensus.cc:2454] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb7a459db0242ec83f65b1d7e56b6a4 in term 0.
I20260620 14:58:18.176826 14465 leader_election.cc:302] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eeb7a459db0242ec83f65b1d7e56b6a4, f270430196cb4484ac1e0eb57fd83b17; no voters:
I20260620 14:58:18.177031 14813 raft_consensus.cc:2781] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:18.177089 14813 raft_consensus.cc:482] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:18.177134 14813 raft_consensus.cc:3037] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:18.178160 14813 raft_consensus.cc:504] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.178542 14813 leader_election.cc:288] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [CANDIDATE]: Term 1 election: Requested vote from peers f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267), ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801)
I20260620 14:58:18.178820 14393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9965d8b0c1d41a880d8ca036933cc32" candidate_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f270430196cb4484ac1e0eb57fd83b17"
I20260620 14:58:18.178951 14393 raft_consensus.cc:3037] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:18.178906 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9965d8b0c1d41a880d8ca036933cc32" candidate_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad147cf6977d42b794ff577c738f277a"
I20260620 14:58:18.179036 14666 raft_consensus.cc:3037] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:18.179915 14393 raft_consensus.cc:2454] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb7a459db0242ec83f65b1d7e56b6a4 in term 1.
I20260620 14:58:18.179950 14666 raft_consensus.cc:2454] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb7a459db0242ec83f65b1d7e56b6a4 in term 1.
I20260620 14:58:18.180181 14465 leader_election.cc:302] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eeb7a459db0242ec83f65b1d7e56b6a4, f270430196cb4484ac1e0eb57fd83b17; no voters:
I20260620 14:58:18.180364 14813 raft_consensus.cc:2781] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:18.180577 14813 raft_consensus.cc:686] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 LEADER]: Becoming Leader. State: Replica: eeb7a459db0242ec83f65b1d7e56b6a4, State: Running, Role: LEADER
I20260620 14:58:18.180794 14813 consensus_queue.cc:237] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } }
I20260620 14:58:18.181288 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.181014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.130:54521)
0620 14:58:18.181157 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.181217 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:58:18.182444 14260 catalog_manager.cc:5495] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb7a459db0242ec83f65b1d7e56b6a4 (127.7.46.130). New cstate: current_term: 1 leader_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeb7a459db0242ec83f65b1d7e56b6a4" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 } health_report { overall_health: UNKNOWN } } }
I20260620 14:58:18.221639 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.218491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.7.46.130:36791)
0620 14:58:18.218637 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:18.218642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:18.218656 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:18.218778 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:18.218781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:18.218837 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:18.218913 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:18.219162 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.219742 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.220620 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:18.220847 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:18.220957 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:18.221208 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:18.221283 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:18.221361 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:18.221451 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:58:18.221503 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:18.223676 14666 raft_consensus.cc:1261] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 1 FOLLOWER]: Refusing update from remote peer eeb7a459db0242ec83f65b1d7e56b6a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:18.223690 14393 raft_consensus.cc:1261] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [term 1 FOLLOWER]: Refusing update from remote peer eeb7a459db0242ec83f65b1d7e56b6a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:18.224179 14816 consensus_queue.cc:1035] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad147cf6977d42b794ff577c738f277a" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:18.224402 14813 consensus_queue.cc:1035] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f270430196cb4484ac1e0eb57fd83b17" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:18.225862 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.225387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.131:46593)
0620 14:58:18.225734 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.225790 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260620 14:58:18.226073 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.225827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.129:44605)
0620 14:58:18.225952 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.226005 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:58:18.228746 14819 mvcc.cc:205] Tried to move back new op lower bound from 7298938872720121856 to 7298938872550100992. Current Snapshot: MvccSnapshot[applied={T|T < 7298938872720121856}]
I20260620 14:58:18.229382 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.229029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.130:43691)
0620 14:58:18.229248 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.229300 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260620 14:58:18.229809 14820 mvcc.cc:205] Tried to move back new op lower bound from 7298938872720121856 to 7298938872550100992. Current Snapshot: MvccSnapshot[applied={T|T < 7298938872720121856}]
I20260620 14:58:18.230005 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.229709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.131:42761)
0620 14:58:18.229872 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.229924 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 14:58:18.232177 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.231696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.129:35913)
0620 14:58:18.232053 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.232116 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300}
I20260620 14:58:18.233031 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.232026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36619 (local address 127.7.46.130:59301)
0620 14:58:18.232894 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.232953 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":801}
I20260620 14:58:18.233336 14260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:18.233501 14260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:18.237386 14260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d00d1e51f2c04f64b06a066b087582d0: Sending DeleteTablet for 3 replicas of tablet c9965d8b0c1d41a880d8ca036933cc32
I20260620 14:58:18.237914 14373 tablet_service.cc:1514] Processing DeleteTablet for tablet c9965d8b0c1d41a880d8ca036933cc32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:18 UTC) from {username='oryx', principal='oryx/127.7.46.188@KRBTEST.COM'} at 127.0.0.1:60088
I20260620 14:58:18.238284 14510 tablet_service.cc:1514] Processing DeleteTablet for tablet c9965d8b0c1d41a880d8ca036933cc32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:18 UTC) from {username='oryx', principal='oryx/127.7.46.188@KRBTEST.COM'} at 127.0.0.1:49002
I20260620 14:58:18.238314 14646 tablet_service.cc:1514] Processing DeleteTablet for tablet c9965d8b0c1d41a880d8ca036933cc32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:18 UTC) from {username='oryx', principal='oryx/127.7.46.188@KRBTEST.COM'} at 127.0.0.1:56762
I20260620 14:58:18.238590 14830 tablet_replica.cc:330] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: stopping tablet replica
I20260620 14:58:18.238601 14828 tablet_replica.cc:330] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17: stopping tablet replica
I20260620 14:58:18.238713 14830 raft_consensus.cc:2229] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:18.238832 14830 raft_consensus.cc:2258] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:18.238894 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 14214
I20260620 14:58:18.240283 14830 ts_tablet_manager.cc:1905] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:18.238713 14828 raft_consensus.cc:2229] T c9965d8b0c1d41a880d8ca036933cc32 P f270430196cb4484ac1e0eb57fd83b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:18.241106 14829 tablet_replica.cc:330] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: stopping tablet replica
I20260620 14:58:18.241256 14829 raft_consensus.cc:2229] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:18.241447 14829 raft_consensus.cc:2258] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:18.242079 14829 ts_tablet_manager.cc:1905] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:18.242659 14830 ts_tablet_manager.cc:1918] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:18.242753 14830 log.cc:1196] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/c9965d8b0c1d41a880d8ca036933cc32
I20260620 14:58:18.243009 14830 ts_tablet_manager.cc:1939] T c9965d8b0c1d41a880d8ca036933cc32 P ad147cf6977d42b794ff577c738f277a: Deleting consensus metadata
I20260620 14:58:18.243609 14247 catalog_manager.cc:4841] TS ad147cf6977d42b794ff577c738f277a (127.7.46.131:42801): tablet c9965d8b0c1d41a880d8ca036933cc32 (table test-table [id=afc1ab1a21354304a4e4336e68d8b5e0]) successfully deleted
I20260620 14:58:18.245174 14829 ts_tablet_manager.cc:1918] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:18.245256 14829 log.cc:1196] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/c9965d8b0c1d41a880d8ca036933cc32
I20260620 14:58:18.245543 14829 ts_tablet_manager.cc:1939] T c9965d8b0c1d41a880d8ca036933cc32 P eeb7a459db0242ec83f65b1d7e56b6a4: Deleting consensus metadata
I20260620 14:58:18.246344 14245 catalog_manager.cc:4841] TS eeb7a459db0242ec83f65b1d7e56b6a4 (127.7.46.130:36791): tablet c9965d8b0c1d41a880d8ca036933cc32 (table test-table [id=afc1ab1a21354304a4e4336e68d8b5e0]) successfully deleted
I20260620 14:58:18.246510 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 14447
I20260620 14:58:18.247253 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.246615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:40267 (local address 127.0.0.1:60092)
0620 14:58:18.247099 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.247163 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:40267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388}
W20260620 14:58:18.247426 14246 catalog_manager.cc:4584] TS f270430196cb4484ac1e0eb57fd83b17 (127.7.46.129:40267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9965d8b0c1d41a880d8ca036933cc32: Network error: Client connection negotiation failed: client connection to 127.7.46.129:40267: connect: Connection refused (error 111)
I20260620 14:58:18.253605 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 14583
I20260620 14:58:18.259075 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 13933
I20260620 14:58:18.264676 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 14041
2026-06-20T14:58:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5233 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:18 dist-test-slave-30t4 krb5kdc[14833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:18 dist-test-slave-30t4 krb5kdc[14833](info): set up 2 sockets
Jun 20 14:58:18 dist-test-slave-30t4 krb5kdc[14833](info): commencing operation
krb5kdc: starting...
W20260620 14:58:19.324399 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:19 dist-test-slave-30t4 krb5kdc[14833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.190.keytab.
Entry for principal oryx/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.190.keytab.
I20260620 14:58:19.439263 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:35921
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.190.keytab
--principal=oryx/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:19.546942 14846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:19.547176 14846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:19.547214 14846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:19.551146 14846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:19.551194 14846 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:58:19.551211 14846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:19.551229 14846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:19.551245 14846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:19.551259 14846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:19.556425 14846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:35921
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.190.keytab
--principal=oryx/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:19.557370 14846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:19.558329 14846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:19.566414 14854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:19.566437 14851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:19.566506 14852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:19.566661 14846 server_base.cc:1034] running on GCE node
I20260620 14:58:19.567296 14846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:19.568506 14846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:19.569741 14846 hybrid_clock.cc:648] HybridClock initialized: now 1781967499569712 us; error 46 us; skew 500 ppm
Jun 20 14:58:19 dist-test-slave-30t4 krb5kdc[14833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967499, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:19.572638 14846 init.cc:377] Logged in from keytab as oryx/127.7.46.190@KRBTEST.COM (short username oryx)
I20260620 14:58:19.573518 14846 webserver.cc:460] Webserver started at http://127.7.46.190:39315/ using document root <none> and password file <none>
I20260620 14:58:19.574059 14846 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:19.574148 14846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:19.574357 14846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:19.576395 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "cc061f65acee41fc96f0c770b44422bd"
format_stamp: "Formatted at 2026-06-20 14:58:19 on dist-test-slave-30t4"
server_key: "3c2a017a8a37267e15094b76940106dc"
server_key_iv: "615d72a26773904e1793f662b51f9bc7"
server_key_version: "encryptionkey@0"
I20260620 14:58:19.576949 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "cc061f65acee41fc96f0c770b44422bd"
format_stamp: "Formatted at 2026-06-20 14:58:19 on dist-test-slave-30t4"
server_key: "3c2a017a8a37267e15094b76940106dc"
server_key_iv: "615d72a26773904e1793f662b51f9bc7"
server_key_version: "encryptionkey@0"
I20260620 14:58:19.580492 14846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:19.582695 14861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:19.583976 14846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:19.584105 14846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "cc061f65acee41fc96f0c770b44422bd"
format_stamp: "Formatted at 2026-06-20 14:58:19 on dist-test-slave-30t4"
server_key: "3c2a017a8a37267e15094b76940106dc"
server_key_iv: "615d72a26773904e1793f662b51f9bc7"
server_key_version: "encryptionkey@0"
I20260620 14:58:19.584208 14846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:19.602468 14846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:19.605144 14846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:19.605340 14846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:19.612941 14846 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:35921
I20260620 14:58:19.612972 14923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:35921 every 8 connection(s)
I20260620 14:58:19.613819 14846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:19.614850 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 14846
I20260620 14:58:19.615001 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:19.615288 7354 external_mini_cluster.cc:1428] Setting key 16002b50a01d0c543f23615cbe2b2cf6
I20260620 14:58:19.616827 14924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:19.623185 14924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: Bootstrap starting.
Jun 20 14:58:19 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:19.625744 14924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:19.626914 14924 log.cc:824] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:19.628942 14924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: No bootstrap required, opened a new log
I20260620 14:58:19.629176 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:19.616573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.7.46.190:35921)
0620 14:58:19.616946 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:19.616960 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:19.616993 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:58:19.617580 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:19.617594 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:19.617890 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:19.618154 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:19.619031 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.620062 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.621181 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.621606 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.621711 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:19.624566 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:19.624571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:19.624583 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:19.624607 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:19.626328 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:19.626773 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:19.626777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:19.626781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:19.626878 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:19.627192 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:19.627194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:19.627196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:19.627677 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:19.627895 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:19.628171 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 14:58:19.628463 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":120,"threads_started":1}
I20260620 14:58:19.632203 14924 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc061f65acee41fc96f0c770b44422bd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35921 } }
I20260620 14:58:19.632398 14924 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:19.632431 14924 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc061f65acee41fc96f0c770b44422bd, State: Initialized, Role: FOLLOWER
I20260620 14:58:19.632929 14924 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc061f65acee41fc96f0c770b44422bd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35921 } }
I20260620 14:58:19.633087 14924 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:19.633152 14924 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:19.633244 14924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:19.634060 14924 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc061f65acee41fc96f0c770b44422bd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35921 } }
I20260620 14:58:19.634379 14924 leader_election.cc:302] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc061f65acee41fc96f0c770b44422bd; no voters:
I20260620 14:58:19.634676 14924 leader_election.cc:288] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:19.634817 14929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:19.635049 14929 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [term 1 LEADER]: Becoming Leader. State: Replica: cc061f65acee41fc96f0c770b44422bd, State: Running, Role: LEADER
I20260620 14:58:19.635358 14929 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc061f65acee41fc96f0c770b44422bd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35921 } }
I20260620 14:58:19.635931 14924 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:19.636987 14930 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc061f65acee41fc96f0c770b44422bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc061f65acee41fc96f0c770b44422bd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35921 } } }
I20260620 14:58:19.637113 14930 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:19.637425 14931 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc061f65acee41fc96f0c770b44422bd. Latest consensus state: current_term: 1 leader_uuid: "cc061f65acee41fc96f0c770b44422bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc061f65acee41fc96f0c770b44422bd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35921 } } }
I20260620 14:58:19.637488 14938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:19.637514 14931 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:19.640501 14938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:19.646085 14938 catalog_manager.cc:1347] Generated new cluster ID: 5e3e80e6368b472a93fffc956b49c65f
I20260620 14:58:19.646143 14938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:19.662739 14938 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:19.663934 14938 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:19.672801 14938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: Generated new TSK 0
I20260620 14:58:19.673522 14938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab.
Entry for principal oryx/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab.
I20260620 14:58:19.700949 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35921
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab
--principal=oryx/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:19.807358 14950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:19.807583 14950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:19.807624 14950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:19.811533 14950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:19.811586 14950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:19.811623 14950 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:58:19.811654 14950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:19.817119 14950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab
--principal=oryx/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:19.818063 14950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:19.819018 14950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:19.828264 14956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:19.828286 14955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:19.828306 14958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:19.828338 14950 server_base.cc:1034] running on GCE node
I20260620 14:58:19.828838 14950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:19.829439 14950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:19.830626 14950 hybrid_clock.cc:648] HybridClock initialized: now 1781967499830598 us; error 43 us; skew 500 ppm
Jun 20 14:58:19 dist-test-slave-30t4 krb5kdc[14833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967499, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:19.833158 14950 init.cc:377] Logged in from keytab as oryx/127.7.46.129@KRBTEST.COM (short username oryx)
I20260620 14:58:19.833915 14950 webserver.cc:460] Webserver started at http://127.7.46.129:45149/ using document root <none> and password file <none>
I20260620 14:58:19.834445 14950 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:19.834554 14950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:19.834767 14950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:19.836823 14950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "885b8ab3500b48b49c56c974e3827a16"
format_stamp: "Formatted at 2026-06-20 14:58:19 on dist-test-slave-30t4"
server_key: "693758bd2a703e1ab2d8246723e57265"
server_key_iv: "c1741ca90a737c268a06384bd4cfc161"
server_key_version: "encryptionkey@0"
I20260620 14:58:19.837419 14950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "885b8ab3500b48b49c56c974e3827a16"
format_stamp: "Formatted at 2026-06-20 14:58:19 on dist-test-slave-30t4"
server_key: "693758bd2a703e1ab2d8246723e57265"
server_key_iv: "c1741ca90a737c268a06384bd4cfc161"
server_key_version: "encryptionkey@0"
I20260620 14:58:19.840867 14950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:19.843053 14965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:19.844182 14950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:19.844307 14950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "885b8ab3500b48b49c56c974e3827a16"
format_stamp: "Formatted at 2026-06-20 14:58:19 on dist-test-slave-30t4"
server_key: "693758bd2a703e1ab2d8246723e57265"
server_key_iv: "c1741ca90a737c268a06384bd4cfc161"
server_key_version: "encryptionkey@0"
I20260620 14:58:19.844419 14950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:19.861398 14950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:19.864032 14950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:19.864226 14950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:19.865619 14950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:19.865695 14950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:19.865760 14950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:19.865813 14950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:58:19 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967499, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:19.876549 14950 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:45205
I20260620 14:58:19.876619 15080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:45205 every 8 connection(s)
I20260620 14:58:19.877440 14950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:19.879278 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:19.866493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:45151 (local address 127.7.46.190:35921)
0620 14:58:19.866660 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:19.866665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:19.867571 (+ 906us) server_negotiation.cc:407] Connection header received
0620 14:58:19.868592 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:19.868596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:19.868660 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:19.868767 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:19.869941 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.870497 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.871735 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.872021 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.872141 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:19.875561 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:19.875565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:19.875568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:19.875593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:19.877188 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:19.877766 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:19.877771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:19.877773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:19.877856 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:19.878225 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:19.878228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:19.878230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:19.878405 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:19.878530 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:19.878992 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 14:58:19.879112 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:58:19.880041 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:19.866957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35921 (local address 127.7.46.129:45151)
0620 14:58:19.867399 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:19.867447 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:19.868342 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:19.868959 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:19.868996 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:19.869325 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:19.869804 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:19.869823 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.870627 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.870630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:19.871610 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:19.871619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.872190 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.872194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:19.872352 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:19.873020 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:19.873041 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:19.875394 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:19.877305 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:19.877312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:19.877329 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:19.877665 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:19.877975 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:19.877979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:19.877980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:19.878135 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:19.878531 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:19.878538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:19.878760 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 14:58:19.878938 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 14:58:19.879237 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
I20260620 14:58:19.881175 15081 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35921
I20260620 14:58:19.881304 15081 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:19.881727 15081 heartbeater.cc:511] Master 127.7.46.190:35921 requested a full tablet report, sending...
I20260620 14:58:19.883250 14877 ts_manager.cc:194] Registered new tserver with Master: 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205)
I20260620 14:58:19.884711 14877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:45151
I20260620 14:58:19.886691 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 14950
I20260620 14:58:19.886828 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:19.887187 7354 external_mini_cluster.cc:1428] Setting key 431d7297005a143098f20e4d09cf584f
I20260620 14:58:19.893606 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:19.883326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:35941 (local address 127.7.46.190:35921)
0620 14:58:19.883498 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:19.883503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:19.883724 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:58:19.883843 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:19.883845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:19.883879 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:19.883947 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:19.884410 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.884949 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.886156 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.886466 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.886570 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:19.889256 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:19.889261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:19.889265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:19.889297 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:19.891205 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:19.891967 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:19.891970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:19.891972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:19.892068 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:19.892793 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:19.892798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:19.892801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:19.893002 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:19.893112 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:19.893366 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:58:19.893468 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:58:19.893620 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:19.883207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35921 (local address 127.7.46.129:35941)
0620 14:58:19.883638 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:19.883652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:19.883743 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:19.883996 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:19.883999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:19.884198 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:19.884307 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:19.884314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.885079 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.885083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:19.886024 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:19.886034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:19.887112 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:19.887120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:19.887244 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:19.888220 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:19.888241 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:19.889078 (+ 837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:19.891395 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:19.891400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:19.891403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:19.891689 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:19.892196 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:19.892201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:19.892204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:19.892662 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:19.893102 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:19.893106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:19.893199 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:58:19.893301 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:58:19.893442 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.7.46.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab.
Entry for principal oryx/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab.
I20260620 14:58:19.925009 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35921
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab
--principal=oryx/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:20.093950 15090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:20.094165 15090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:20.094208 15090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:20.098163 15090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:20.098212 15090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:20.098248 15090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:58:20.098353 15090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:20.103394 15090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab
--principal=oryx/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:20.104390 15090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:20.105331 15090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:20.114519 15096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:20.114567 15098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:20.114514 15095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:20.114992 15090 server_base.cc:1034] running on GCE node
I20260620 14:58:20.115440 15090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:20.116101 15090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:20.117292 15090 hybrid_clock.cc:648] HybridClock initialized: now 1781967500117267 us; error 59 us; skew 500 ppm
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967500, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:20.120026 15090 init.cc:377] Logged in from keytab as oryx/127.7.46.130@KRBTEST.COM (short username oryx)
I20260620 14:58:20.120924 15090 webserver.cc:460] Webserver started at http://127.7.46.130:37115/ using document root <none> and password file <none>
I20260620 14:58:20.121495 15090 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:20.121553 15090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:20.121757 15090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:20.123684 15090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "009a44f941f343219a27a80ea0ee23b1"
format_stamp: "Formatted at 2026-06-20 14:58:20 on dist-test-slave-30t4"
server_key: "3bfbba970aa5f2120c03b6aa596b51d9"
server_key_iv: "f68be189f59af85071f2b5b077c58f44"
server_key_version: "encryptionkey@0"
I20260620 14:58:20.124269 15090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "009a44f941f343219a27a80ea0ee23b1"
format_stamp: "Formatted at 2026-06-20 14:58:20 on dist-test-slave-30t4"
server_key: "3bfbba970aa5f2120c03b6aa596b51d9"
server_key_iv: "f68be189f59af85071f2b5b077c58f44"
server_key_version: "encryptionkey@0"
I20260620 14:58:20.128187 15090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:58:20.130645 15105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:20.131991 15090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:20.132122 15090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "009a44f941f343219a27a80ea0ee23b1"
format_stamp: "Formatted at 2026-06-20 14:58:20 on dist-test-slave-30t4"
server_key: "3bfbba970aa5f2120c03b6aa596b51d9"
server_key_iv: "f68be189f59af85071f2b5b077c58f44"
server_key_version: "encryptionkey@0"
I20260620 14:58:20.132226 15090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:20.150137 15090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:20.153321 15090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:20.153523 15090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:20.155009 15090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:20.155079 15090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:20.155143 15090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:20.155195 15090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967500, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:20.166324 15090 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:39083
I20260620 14:58:20.166474 15220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:39083 every 8 connection(s)
I20260620 14:58:20.167186 15090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:20.169770 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.156157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:55925 (local address 127.7.46.190:35921)
0620 14:58:20.156318 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.156324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.157408 (+ 1084us) server_negotiation.cc:407] Connection header received
0620 14:58:20.158934 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.158940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.159009 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.159072 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:20.160933 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.161741 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.163222 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.163494 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.163617 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.166278 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:20.166282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:20.166285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:20.166322 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:20.168083 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.168525 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.168528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.168529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.168605 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.168901 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.168903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.168904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.169077 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:20.169154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.169563 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.169655 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:58:20.170509 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.156674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35921 (local address 127.7.46.130:55925)
0620 14:58:20.157228 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.157286 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.158661 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.159516 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.159528 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.160074 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:20.160767 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.160785 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.161893 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.161896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.163105 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.163117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.163654 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.163658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.163904 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.164512 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:20.164537 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:20.166073 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:20.168185 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.168189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.168199 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.168436 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.168686 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.168688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.168690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.168822 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.169163 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:20.169167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:20.169372 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.169518 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.169734 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":144,"threads_started":1}
I20260620 14:58:20.171646 15221 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35921
I20260620 14:58:20.171837 15221 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:20.172021 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 15090
I20260620 14:58:20.172143 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:20.172386 15221 heartbeater.cc:511] Master 127.7.46.190:35921 requested a full tablet report, sending...
I20260620 14:58:20.172472 7354 external_mini_cluster.cc:1428] Setting key 11d190bd208fd83826299c8073417bf3
I20260620 14:58:20.173537 14877 ts_manager.cc:194] Registered new tserver with Master: 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083)
I20260620 14:58:20.174306 14877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:55925
I20260620 14:58:20.182305 15227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.173953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35921 (local address 127.7.46.130:39341)
0620 14:58:20.174232 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.174245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.174380 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.174806 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.174811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.175033 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:20.175181 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.175191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.176192 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.176197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.177295 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.177309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.177772 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.177779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.177909 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.178458 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:20.178478 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:20.178762 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:20.180795 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.180797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.180799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.180990 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.181280 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.181282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.181283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.181357 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.181852 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:20.181854 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:20.181924 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.182017 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.182115 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260620 14:58:20.182408 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.173999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:39341 (local address 127.7.46.190:35921)
0620 14:58:20.174091 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.174094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.174303 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:58:20.174511 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.174517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.174571 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.174665 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:20.175324 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.176051 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.177434 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.177650 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.177735 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.178905 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:20.178912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:20.178918 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:20.178947 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:20.180693 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.181095 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.181101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.181102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.181193 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.181459 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.181463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.181464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.181748 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:20.181847 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.182115 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.182229 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for oryx/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab.
Entry for principal oryx/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab.
I20260620 14:58:20.200187 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35921
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab
--principal=oryx/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:20.308692 15230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:20.308931 15230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:20.308972 15230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:20.312918 15230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:20.312968 15230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:20.313031 15230 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:58:20.313108 15230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:20.318214 15230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab
--principal=oryx/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.46.190:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:20.319186 15230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:20.320212 15230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:20.329645 15236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:20.329659 15238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:20.329614 15235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:20.329692 15230 server_base.cc:1034] running on GCE node
I20260620 14:58:20.330281 15230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:20.330905 15230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:20.332113 15230 hybrid_clock.cc:648] HybridClock initialized: now 1781967500332102 us; error 28 us; skew 500 ppm
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967500, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:20.334638 15230 init.cc:377] Logged in from keytab as oryx/127.7.46.131@KRBTEST.COM (short username oryx)
I20260620 14:58:20.335510 15230 webserver.cc:460] Webserver started at http://127.7.46.131:44291/ using document root <none> and password file <none>
I20260620 14:58:20.336107 15230 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:20.336198 15230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:20.336460 15230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:20.338486 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "24ca40ce1631454d8b4a3cf5193b056c"
format_stamp: "Formatted at 2026-06-20 14:58:20 on dist-test-slave-30t4"
server_key: "4a660d578472c3f86fdad7aa2d6bf8b2"
server_key_iv: "6960b0097483fa5760991df690b4386a"
server_key_version: "encryptionkey@0"
I20260620 14:58:20.339049 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "24ca40ce1631454d8b4a3cf5193b056c"
format_stamp: "Formatted at 2026-06-20 14:58:20 on dist-test-slave-30t4"
server_key: "4a660d578472c3f86fdad7aa2d6bf8b2"
server_key_iv: "6960b0097483fa5760991df690b4386a"
server_key_version: "encryptionkey@0"
I20260620 14:58:20.342651 15230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:58:20.344986 15245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:20.346129 15230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:20.346267 15230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "24ca40ce1631454d8b4a3cf5193b056c"
format_stamp: "Formatted at 2026-06-20 14:58:20 on dist-test-slave-30t4"
server_key: "4a660d578472c3f86fdad7aa2d6bf8b2"
server_key_iv: "6960b0097483fa5760991df690b4386a"
server_key_version: "encryptionkey@0"
I20260620 14:58:20.346374 15230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:20.365522 15230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:20.368218 15230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:20.368412 15230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:20.369714 15230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:20.369789 15230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:20.369853 15230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:20.369886 15230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967500, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:20.380872 15230 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:35907
I20260620 14:58:20.380946 15360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:35907 every 8 connection(s)
I20260620 14:58:20.381884 15230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:20.383406 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.370592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:52327 (local address 127.7.46.190:35921)
0620 14:58:20.370718 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.370722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.371815 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 14:58:20.372916 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.372920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.372984 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.373068 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:20.374836 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.375571 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.376724 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.376956 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.377077 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.379814 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:20.379817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:20.379820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:20.379849 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:20.381454 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.381946 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.381951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.381953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.382036 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.382412 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.382416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.382420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.382658 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:20.382763 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.383163 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.383293 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:20.384179 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.371144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35921 (local address 127.7.46.131:52327)
0620 14:58:20.371635 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.371676 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.372675 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.373528 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.373538 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.373967 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:20.374696 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.374719 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.375679 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.375682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.376593 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.376603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.377122 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.377126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.377299 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.377881 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:20.377900 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:20.379630 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:20.381580 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.381585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.381598 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.381838 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.382134 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.382139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.382141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.382315 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.382768 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:20.382773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:20.382970 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.383123 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.383340 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":145,"threads_started":1}
I20260620 14:58:20.385300 15361 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35921
I20260620 14:58:20.385439 15361 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:20.385865 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 15230
I20260620 14:58:20.385947 15361 heartbeater.cc:511] Master 127.7.46.190:35921 requested a full tablet report, sending...
I20260620 14:58:20.386014 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:20.386355 7354 external_mini_cluster.cc:1428] Setting key 604c277dae58e9d245f0fd800741d298
I20260620 14:58:20.387055 14877 ts_manager.cc:194] Registered new tserver with Master: 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:20.387850 14877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:52327
I20260620 14:58:20.389356 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:20.397588 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.387469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35921 (local address 127.7.46.131:36825)
0620 14:58:20.388417 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.388431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.389015 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.391022 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.391024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.391176 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:20.391259 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.391263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.392052 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.392055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.392602 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.392609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.393149 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.393152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.393244 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.393725 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:20.393736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:20.394029 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:20.396099 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.396103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.396104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.396284 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.396578 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.396580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.396582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.396650 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.397120 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:20.397126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:20.397205 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.397288 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.397382 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":88,"threads_started":1}
I20260620 14:58:20.397657 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.387546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:36825 (local address 127.7.46.190:35921)
0620 14:58:20.387661 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.387664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.388515 (+ 851us) server_negotiation.cc:407] Connection header received
0620 14:58:20.389158 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.389161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.389209 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.389254 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:20.391372 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.391928 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.392722 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.393037 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.393145 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.394187 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:20.394190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:20.394193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:20.394215 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:20.395927 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.396385 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.396391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.396396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.396485 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.396740 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.396746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.396751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.397009 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:20.397119 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.397380 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.397497 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:58:20.412312 14927 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:20.406729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.7.46.190:35921)
0620 14:58:20.406842 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.406846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.406857 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:58:20.406889 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.406891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.406922 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.406989 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:20.407634 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.408179 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.408956 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.409156 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.409225 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.412151 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46910: BlockingRecv error: recv got EOF from 127.0.0.1:46910 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:20.420557 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.413560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.7.46.190:35921)
0620 14:58:20.413666 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.413669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.413753 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:58:20.413839 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.413841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.413871 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.413933 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:20.414441 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.414898 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.415596 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.415861 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.415936 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.417910 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:20.417913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:20.417915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:20.417935 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:20.419149 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.419508 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.419511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.419512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.419580 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.419843 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.419845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.419846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.419995 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:20.420093 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.420277 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.420357 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:58:20.423094 14877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:20.426044 14877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:20.439595 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.434025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:39083 (local address 127.0.0.1:58094)
0620 14:58:20.434629 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.434643 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.434720 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.435604 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.435607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.435626 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.435727 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.435732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.437176 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.437180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.438269 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.438282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.439054 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.439058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.439170 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.439208 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.439343 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.439461 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":25,"thread_start_us":96,"threads_started":1}
I20260620 14:58:20.439606 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.434599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.7.46.130:39083)
0620 14:58:20.435111 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.435122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.435142 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:58:20.435212 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.435216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.435391 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.435897 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.436048 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.437068 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.438405 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.438930 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.439056 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.439215 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.439371 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.439464 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":106,"threads_started":1}
I20260620 14:58:20.439860 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.433808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45205 (local address 127.0.0.1:37026)
0620 14:58:20.434130 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.434178 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.434301 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.435079 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.435085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.435108 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.435224 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.435231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.436611 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.436615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.438122 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.438132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.439417 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.439422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.439519 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.439544 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.439623 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.439690 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":21,"thread_start_us":87,"threads_started":1}
I20260620 14:58:20.440217 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.433946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.7.46.129:45205)
0620 14:58:20.434311 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.434318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.434341 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:58:20.434525 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.434532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.434749 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.434890 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.435348 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.436504 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.438282 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.439085 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.439187 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.439349 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.439962 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.440071 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":70,"threads_started":1}
I20260620 14:58:20.440764 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.434528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35907 (local address 127.0.0.1:44836)
0620 14:58:20.435560 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.435575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.435683 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.436209 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.436213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.436269 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.436357 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.436365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.437849 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.437853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.439057 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.439069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.440354 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.440358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.440446 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.440471 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.440550 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.440619 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":27,"thread_start_us":89,"threads_started":1}
I20260620 14:58:20.441183 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.434947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.7.46.131:35907)
0620 14:58:20.435371 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.435387 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.435674 (+ 287us) server_negotiation.cc:407] Connection header received
0620 14:58:20.435828 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.435839 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.436056 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.436219 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.436485 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.437693 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.439211 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.440023 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.440130 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.440281 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.440945 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.441038 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":99,"threads_started":1}
I20260620 14:58:20.442701 15155 tablet_service.cc:1467] Processing CreateTablet for tablet 394d21e4296f4772905434959685a4b1 (DEFAULT_TABLE table=test-table [id=4139b41adff14c108ed33e6a67e56376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:20.442793 15015 tablet_service.cc:1467] Processing CreateTablet for tablet 394d21e4296f4772905434959685a4b1 (DEFAULT_TABLE table=test-table [id=4139b41adff14c108ed33e6a67e56376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:20.443653 15155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394d21e4296f4772905434959685a4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.443815 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394d21e4296f4772905434959685a4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.443737 15295 tablet_service.cc:1467] Processing CreateTablet for tablet 394d21e4296f4772905434959685a4b1 (DEFAULT_TABLE table=test-table [id=4139b41adff14c108ed33e6a67e56376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:20.444636 15295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394d21e4296f4772905434959685a4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.450260 15389 tablet_bootstrap.cc:492] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Bootstrap starting.
I20260620 14:58:20.451758 15390 tablet_bootstrap.cc:492] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Bootstrap starting.
I20260620 14:58:20.451859 15391 tablet_bootstrap.cc:492] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Bootstrap starting.
I20260620 14:58:20.452986 15389 tablet_bootstrap.cc:654] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.453830 15390 tablet_bootstrap.cc:654] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.453850 15389 log.cc:824] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:20.453966 15391 tablet_bootstrap.cc:654] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.454699 15391 log.cc:824] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:20.455256 15390 log.cc:824] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:20.456061 15389 tablet_bootstrap.cc:492] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: No bootstrap required, opened a new log
I20260620 14:58:20.456249 15389 ts_tablet_manager.cc:1397] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:58:20.456622 15391 tablet_bootstrap.cc:492] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: No bootstrap required, opened a new log
I20260620 14:58:20.456794 15391 ts_tablet_manager.cc:1397] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:58:20.457060 15390 tablet_bootstrap.cc:492] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: No bootstrap required, opened a new log
I20260620 14:58:20.457258 15390 ts_tablet_manager.cc:1397] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:58:20.460198 15389 raft_consensus.cc:348] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.460459 15389 raft_consensus.cc:374] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.460556 15389 raft_consensus.cc:729] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009a44f941f343219a27a80ea0ee23b1, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.460387 15390 raft_consensus.cc:348] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.460496 15391 raft_consensus.cc:348] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.460718 15390 raft_consensus.cc:374] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.460755 15391 raft_consensus.cc:374] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.460824 15390 raft_consensus.cc:729] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885b8ab3500b48b49c56c974e3827a16, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.460845 15391 raft_consensus.cc:729] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ca40ce1631454d8b4a3cf5193b056c, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.461024 15389 consensus_queue.cc:260] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.461217 15391 consensus_queue.cc:260] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.461344 15390 consensus_queue.cc:260] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.461714 15221 heartbeater.cc:503] Master 127.7.46.190:35921 was elected leader, sending a full tablet report...
I20260620 14:58:20.462042 15389 ts_tablet_manager.cc:1428] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:58:20.462069 15391 ts_tablet_manager.cc:1428] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:20.462371 15361 heartbeater.cc:503] Master 127.7.46.190:35921 was elected leader, sending a full tablet report...
I20260620 14:58:20.462371 15390 ts_tablet_manager.cc:1428] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 14:58:20.462692 15081 heartbeater.cc:503] Master 127.7.46.190:35921 was elected leader, sending a full tablet report...
I20260620 14:58:20.602749 15395 raft_consensus.cc:482] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:20.602995 15395 raft_consensus.cc:504] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.604259 15395 leader_election.cc:288] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205), 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:20.607939 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.604501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45205 (local address 127.7.46.130:40207)
0620 14:58:20.604634 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.604647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.604755 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.605127 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.605130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.605153 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.605240 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.605244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.605977 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.605983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.606757 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.606765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.607446 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.607450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.607591 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.607608 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.607700 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.607772 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":24}
I20260620 14:58:20.607939 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.604593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:40207 (local address 127.7.46.129:45205)
0620 14:58:20.604736 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.604740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.604757 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:58:20.604861 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.604864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.604926 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.605010 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.605368 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.605861 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.606899 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.607321 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.607457 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.607513 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.607716 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.607818 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:20.608579 15035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394d21e4296f4772905434959685a4b1" candidate_uuid: "009a44f941f343219a27a80ea0ee23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885b8ab3500b48b49c56c974e3827a16" is_pre_election: true
I20260620 14:58:20.608796 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.604953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:50441 (local address 127.7.46.131:35907)
0620 14:58:20.605086 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.605090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.605261 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:58:20.605405 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.605408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.605455 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.605540 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.606161 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.606651 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.607674 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.608255 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.608385 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.608430 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.608626 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.608687 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:58:20.608825 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.604786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35907 (local address 127.7.46.130:50441)
0620 14:58:20.605172 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.605182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.605291 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.605564 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.605567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.605582 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.606033 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.606039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.606764 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.606770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.607547 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.607555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.608385 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.608388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.608497 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.608514 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.608609 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.608677 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":22,"thread_start_us":170,"threads_started":1}
I20260620 14:58:20.608911 15035 raft_consensus.cc:2454] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009a44f941f343219a27a80ea0ee23b1 in term 0.
I20260620 14:58:20.609383 15315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394d21e4296f4772905434959685a4b1" candidate_uuid: "009a44f941f343219a27a80ea0ee23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca40ce1631454d8b4a3cf5193b056c" is_pre_election: true
I20260620 14:58:20.609412 15106 leader_election.cc:302] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009a44f941f343219a27a80ea0ee23b1, 885b8ab3500b48b49c56c974e3827a16; no voters:
I20260620 14:58:20.609663 15315 raft_consensus.cc:2454] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009a44f941f343219a27a80ea0ee23b1 in term 0.
I20260620 14:58:20.609678 15395 raft_consensus.cc:2781] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:20.609755 15395 raft_consensus.cc:482] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:20.609807 15395 raft_consensus.cc:3037] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.610931 15395 raft_consensus.cc:504] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.611304 15395 leader_election.cc:288] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [CANDIDATE]: Term 1 election: Requested vote from peers 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205), 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:20.611644 15035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394d21e4296f4772905434959685a4b1" candidate_uuid: "009a44f941f343219a27a80ea0ee23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885b8ab3500b48b49c56c974e3827a16"
I20260620 14:58:20.611654 15315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394d21e4296f4772905434959685a4b1" candidate_uuid: "009a44f941f343219a27a80ea0ee23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca40ce1631454d8b4a3cf5193b056c"
I20260620 14:58:20.611742 15035 raft_consensus.cc:3037] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.611796 15315 raft_consensus.cc:3037] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.612952 15315 raft_consensus.cc:2454] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009a44f941f343219a27a80ea0ee23b1 in term 1.
I20260620 14:58:20.612963 15035 raft_consensus.cc:2454] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009a44f941f343219a27a80ea0ee23b1 in term 1.
I20260620 14:58:20.613251 15106 leader_election.cc:302] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009a44f941f343219a27a80ea0ee23b1, 24ca40ce1631454d8b4a3cf5193b056c; no voters:
I20260620 14:58:20.613420 15395 raft_consensus.cc:2781] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:20.613687 15395 raft_consensus.cc:686] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 1 LEADER]: Becoming Leader. State: Replica: 009a44f941f343219a27a80ea0ee23b1, State: Running, Role: LEADER
I20260620 14:58:20.614006 15395 consensus_queue.cc:237] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } }
I20260620 14:58:20.620280 14878 catalog_manager.cc:5495] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130). New cstate: current_term: 1 leader_uuid: "009a44f941f343219a27a80ea0ee23b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } health_report { overall_health: HEALTHY } } }
W20260620 14:58:20.628698 15082 tablet.cc:2369] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:20.633383 15362 tablet.cc:2369] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:20.668468 15222 tablet.cc:2369] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:20.675612 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.672375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.7.46.130:39083)
0620 14:58:20.672501 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.672505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.672538 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:58:20.672694 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.672696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.672749 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.672832 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:20.673085 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.673578 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.674360 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.674570 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.674643 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.674863 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:20.675256 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:20.675338 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.675398 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.675444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:58:20.680887 15035 raft_consensus.cc:1261] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Refusing update from remote peer 009a44f941f343219a27a80ea0ee23b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:20.680891 15315 raft_consensus.cc:1261] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Refusing update from remote peer 009a44f941f343219a27a80ea0ee23b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:20.681555 15395 consensus_queue.cc:1035] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:20.681759 15399 consensus_queue.cc:1035] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:20.692128 15402 mvcc.cc:205] Tried to move back new op lower bound from 7298938882782113792 to 7298938882517553152. Current Snapshot: MvccSnapshot[applied={T|T < 7298938882782113792 or (T in {7298938882782113792})}]
I20260620 14:58:20.696828 14877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:20.696970 14877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:20.699332 14877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: Sending DeleteTablet for 3 replicas of tablet 394d21e4296f4772905434959685a4b1
I20260620 14:58:20.700127 15155 tablet_service.cc:1514] Processing DeleteTablet for tablet 394d21e4296f4772905434959685a4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:20 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:58094
I20260620 14:58:20.700139 15015 tablet_service.cc:1514] Processing DeleteTablet for tablet 394d21e4296f4772905434959685a4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:20 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:37026
I20260620 14:58:20.700579 15295 tablet_service.cc:1514] Processing DeleteTablet for tablet 394d21e4296f4772905434959685a4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:20 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:44836
I20260620 14:58:20.700647 15412 tablet_replica.cc:330] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: stopping tablet replica
I20260620 14:58:20.700994 15412 raft_consensus.cc:2229] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:20.701318 15412 raft_consensus.cc:2258] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:20.701397 15413 tablet_replica.cc:330] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: stopping tablet replica
I20260620 14:58:20.701694 15413 raft_consensus.cc:2229] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:20.701975 15413 raft_consensus.cc:2258] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:20.701665 14877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:20.702322 14877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:20.702464 15411 tablet_replica.cc:330] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: stopping tablet replica
I20260620 14:58:20.702708 15411 raft_consensus.cc:2229] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:20.702993 15411 raft_consensus.cc:2258] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:20.704506 15413 ts_tablet_manager.cc:1905] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:20.704151 15412 ts_tablet_manager.cc:1905] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:20.707093 15015 tablet_service.cc:1467] Processing CreateTablet for tablet ab01a17f90a34977b57390e73577d546 (DEFAULT_TABLE table=test-table [id=d7ecfb9f978c40e7a6676d8069648a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:20.707374 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab01a17f90a34977b57390e73577d546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.708053 15411 ts_tablet_manager.cc:1905] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:20.709158 15412 ts_tablet_manager.cc:1918] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:20.709244 15412 log.cc:1196] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/394d21e4296f4772905434959685a4b1
I20260620 14:58:20.709549 15412 ts_tablet_manager.cc:1939] T 394d21e4296f4772905434959685a4b1 P 885b8ab3500b48b49c56c974e3827a16: Deleting consensus metadata
I20260620 14:58:20.709919 15390 tablet_bootstrap.cc:492] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: Bootstrap starting.
I20260620 14:58:20.710088 15155 tablet_service.cc:1467] Processing CreateTablet for tablet ab01a17f90a34977b57390e73577d546 (DEFAULT_TABLE table=test-table [id=d7ecfb9f978c40e7a6676d8069648a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:20.710318 15155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab01a17f90a34977b57390e73577d546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.710736 15411 ts_tablet_manager.cc:1918] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:20.710834 15411 log.cc:1196] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/394d21e4296f4772905434959685a4b1
I20260620 14:58:20.711140 15390 tablet_bootstrap.cc:654] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.711141 15411 ts_tablet_manager.cc:1939] T 394d21e4296f4772905434959685a4b1 P 009a44f941f343219a27a80ea0ee23b1: Deleting consensus metadata
I20260620 14:58:20.712370 14863 catalog_manager.cc:4841] TS 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083): tablet 394d21e4296f4772905434959685a4b1 (table test-table [id=4139b41adff14c108ed33e6a67e56376]) successfully deleted
I20260620 14:58:20.712556 15295 tablet_service.cc:1467] Processing CreateTablet for tablet ab01a17f90a34977b57390e73577d546 (DEFAULT_TABLE table=test-table [id=d7ecfb9f978c40e7a6676d8069648a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:20.712770 15389 tablet_bootstrap.cc:492] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: Bootstrap starting.
I20260620 14:58:20.712803 15295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab01a17f90a34977b57390e73577d546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.713693 15389 tablet_bootstrap.cc:654] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.713955 15413 ts_tablet_manager.cc:1918] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:20.714042 15413 log.cc:1196] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/394d21e4296f4772905434959685a4b1
I20260620 14:58:20.714335 15413 ts_tablet_manager.cc:1939] T 394d21e4296f4772905434959685a4b1 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting consensus metadata
I20260620 14:58:20.715039 15391 tablet_bootstrap.cc:492] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: Bootstrap starting.
I20260620 14:58:20.715664 14862 catalog_manager.cc:4841] TS 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907): tablet 394d21e4296f4772905434959685a4b1 (table test-table [id=4139b41adff14c108ed33e6a67e56376]) successfully deleted
I20260620 14:58:20.715945 15391 tablet_bootstrap.cc:654] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.716472 15390 tablet_bootstrap.cc:492] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: No bootstrap required, opened a new log
I20260620 14:58:20.716550 15390 ts_tablet_manager.cc:1397] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260620 14:58:20.716673 15389 tablet_bootstrap.cc:492] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: No bootstrap required, opened a new log
I20260620 14:58:20.716739 15389 ts_tablet_manager.cc:1397] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:58:20.717011 15390 raft_consensus.cc:348] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.717121 15390 raft_consensus.cc:374] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.717161 15390 raft_consensus.cc:729] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885b8ab3500b48b49c56c974e3827a16, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.717132 15389 raft_consensus.cc:348] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.717226 15389 raft_consensus.cc:374] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.717269 15389 raft_consensus.cc:729] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009a44f941f343219a27a80ea0ee23b1, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.717288 15390 consensus_queue.cc:260] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.717381 15389 consensus_queue.cc:260] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.717536 15390 ts_tablet_manager.cc:1428] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:58:20.717609 15389 ts_tablet_manager.cc:1428] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:20.718067 14862 catalog_manager.cc:4841] TS 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205): tablet 394d21e4296f4772905434959685a4b1 (table test-table [id=4139b41adff14c108ed33e6a67e56376]) successfully deleted
I20260620 14:58:20.719084 15391 tablet_bootstrap.cc:492] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: No bootstrap required, opened a new log
I20260620 14:58:20.719165 15391 ts_tablet_manager.cc:1397] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:58:20.719584 15391 raft_consensus.cc:348] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.719681 15391 raft_consensus.cc:374] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.719710 15391 raft_consensus.cc:729] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ca40ce1631454d8b4a3cf5193b056c, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.719856 15391 consensus_queue.cc:260] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.720098 15391 ts_tablet_manager.cc:1428] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:20.750672 15397 raft_consensus.cc:482] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:20.750811 15397 raft_consensus.cc:504] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.752197 15397 leader_election.cc:288] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083), 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:20.757743 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.752519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35907 (local address 127.7.46.129:50425)
0620 14:58:20.753604 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.753623 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.753777 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.754122 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.754125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.754146 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.754304 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.754312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.755354 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.755359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.756464 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.756474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.757289 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.757293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.757403 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.757431 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.757507 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.757575 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"spinlock_wait_cycles":5120,"thread_start_us":97,"threads_started":1}
I20260620 14:58:20.757748 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.753737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:50425 (local address 127.7.46.131:35907)
0620 14:58:20.753885 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.753891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.753905 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:20.753965 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.753968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.754008 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.754106 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.754438 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.755230 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.756612 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.757153 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.757275 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.757325 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.757544 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.757621 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:58:20.758195 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.752892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:53107 (local address 127.7.46.130:39083)
0620 14:58:20.753021 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.753026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.754108 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 14:58:20.754311 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.754316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.754375 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.754454 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.754806 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.755354 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.756870 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.757457 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.757593 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.757643 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.758007 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.758087 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:58:20.758203 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.752823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:39083 (local address 127.7.46.129:53107)
0620 14:58:20.754032 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.754045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.754183 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.754566 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.754568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.754582 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.754667 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.754673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.755482 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.755485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.756728 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.756739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.757702 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.757706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.757828 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.757851 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.757983 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.758060 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"spinlock_wait_cycles":896,"thread_start_us":165,"threads_started":1}
I20260620 14:58:20.758755 15315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab01a17f90a34977b57390e73577d546" candidate_uuid: "885b8ab3500b48b49c56c974e3827a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca40ce1631454d8b4a3cf5193b056c" is_pre_election: true
I20260620 14:58:20.758873 15315 raft_consensus.cc:2454] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 885b8ab3500b48b49c56c974e3827a16 in term 0.
I20260620 14:58:20.759241 14966 leader_election.cc:302] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ca40ce1631454d8b4a3cf5193b056c, 885b8ab3500b48b49c56c974e3827a16; no voters:
I20260620 14:58:20.759516 15397 raft_consensus.cc:2781] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:20.759469 15175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab01a17f90a34977b57390e73577d546" candidate_uuid: "885b8ab3500b48b49c56c974e3827a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009a44f941f343219a27a80ea0ee23b1" is_pre_election: true
I20260620 14:58:20.759567 15397 raft_consensus.cc:482] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:20.759613 15397 raft_consensus.cc:3037] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.759675 15175 raft_consensus.cc:2454] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 885b8ab3500b48b49c56c974e3827a16 in term 0.
I20260620 14:58:20.760674 15397 raft_consensus.cc:504] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.761047 15397 leader_election.cc:288] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [CANDIDATE]: Term 1 election: Requested vote from peers 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083), 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:20.761366 15315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab01a17f90a34977b57390e73577d546" candidate_uuid: "885b8ab3500b48b49c56c974e3827a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca40ce1631454d8b4a3cf5193b056c"
I20260620 14:58:20.761497 15315 raft_consensus.cc:3037] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.762298 15315 raft_consensus.cc:2454] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885b8ab3500b48b49c56c974e3827a16 in term 1.
I20260620 14:58:20.762578 15175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab01a17f90a34977b57390e73577d546" candidate_uuid: "885b8ab3500b48b49c56c974e3827a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009a44f941f343219a27a80ea0ee23b1"
I20260620 14:58:20.762609 14966 leader_election.cc:302] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ca40ce1631454d8b4a3cf5193b056c, 885b8ab3500b48b49c56c974e3827a16; no voters:
I20260620 14:58:20.762722 15175 raft_consensus.cc:3037] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.763736 15175 raft_consensus.cc:2454] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885b8ab3500b48b49c56c974e3827a16 in term 1.
I20260620 14:58:20.763988 15397 raft_consensus.cc:2781] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:20.764206 15397 raft_consensus.cc:686] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 1 LEADER]: Becoming Leader. State: Replica: 885b8ab3500b48b49c56c974e3827a16, State: Running, Role: LEADER
I20260620 14:58:20.764453 15397 consensus_queue.cc:237] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.766744 14878 catalog_manager.cc:5495] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129). New cstate: current_term: 1 leader_uuid: "885b8ab3500b48b49c56c974e3827a16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:58:20 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967499, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.190@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:20.806175 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.797354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.7.46.190:35921)
0620 14:58:20.797477 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.797481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.797640 (+ 159us) server_negotiation.cc:407] Connection header received
0620 14:58:20.797720 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.797723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.797768 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.797857 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:20.798461 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.799285 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.800168 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.800500 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.800610 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.802891 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:20.802894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:20.802896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:20.802916 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:20.804394 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.804918 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.804920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.804922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.805008 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:20.805292 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:20.805294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:20.805296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:20.805454 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:20.805527 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.805859 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.806058 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":51}
I20260620 14:58:20.806186 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.797262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35921 (local address 127.0.0.1:46928)
0620 14:58:20.797532 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.797549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.797649 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.797882 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.797884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.798245 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:20.798346 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.798352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.799402 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.799404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.800052 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.800061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.800605 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.800607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.800675 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.801255 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:20.801267 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:20.802754 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:20.804509 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.804519 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.804525 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.804824 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.805100 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:20.805103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:20.805105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:20.805206 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:20.805558 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:20.805567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:20.805692 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.805815 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.805979 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
I20260620 14:58:20.810035 14878 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:46928:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:58:20.810746 14878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:20.815868 15015 tablet_service.cc:1467] Processing CreateTablet for tablet 30b4284ed90c4b00a5bf8860c4d9220d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c916b2006b83452f92d0c485d19fd6de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:58:20.816025 15155 tablet_service.cc:1467] Processing CreateTablet for tablet 30b4284ed90c4b00a5bf8860c4d9220d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c916b2006b83452f92d0c485d19fd6de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:58:20.816257 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4284ed90c4b00a5bf8860c4d9220d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.816332 15155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4284ed90c4b00a5bf8860c4d9220d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.819003 15389 tablet_bootstrap.cc:492] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: Bootstrap starting.
I20260620 14:58:20.819240 15295 tablet_service.cc:1467] Processing CreateTablet for tablet 30b4284ed90c4b00a5bf8860c4d9220d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c916b2006b83452f92d0c485d19fd6de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:58:20.819550 15295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4284ed90c4b00a5bf8860c4d9220d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:20.820026 15389 tablet_bootstrap.cc:654] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.821357 15389 tablet_bootstrap.cc:492] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: No bootstrap required, opened a new log
I20260620 14:58:20.821450 15389 ts_tablet_manager.cc:1397] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:20.821893 15389 raft_consensus.cc:348] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.822027 15389 raft_consensus.cc:374] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.822062 15389 raft_consensus.cc:729] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009a44f941f343219a27a80ea0ee23b1, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.822079 15391 tablet_bootstrap.cc:492] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: Bootstrap starting.
I20260620 14:58:20.822181 15389 consensus_queue.cc:260] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.822476 15389 ts_tablet_manager.cc:1428] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:58:20.822415 15399 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.822585 15399 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:58:20.823014 15391 tablet_bootstrap.cc:654] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.823165 15390 tablet_bootstrap.cc:492] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: Bootstrap starting.
I20260620 14:58:20.824177 15390 tablet_bootstrap.cc:654] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:20.825357 15391 tablet_bootstrap.cc:492] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: No bootstrap required, opened a new log
I20260620 14:58:20.825354 15390 tablet_bootstrap.cc:492] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: No bootstrap required, opened a new log
I20260620 14:58:20.825433 15390 ts_tablet_manager.cc:1397] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:20.825433 15391 ts_tablet_manager.cc:1397] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:58:20.825848 15390 raft_consensus.cc:348] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.825848 15391 raft_consensus.cc:348] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.825966 15391 raft_consensus.cc:374] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.825953 15390 raft_consensus.cc:374] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:20.825999 15391 raft_consensus.cc:729] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ca40ce1631454d8b4a3cf5193b056c, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.825999 15390 raft_consensus.cc:729] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885b8ab3500b48b49c56c974e3827a16, State: Initialized, Role: FOLLOWER
I20260620 14:58:20.826123 15391 consensus_queue.cc:260] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.826135 15390 consensus_queue.cc:260] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.826380 15390 ts_tablet_manager.cc:1428] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:20.826371 15396 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.826541 15396 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:58:20.826649 15397 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.826763 15397 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:58:20.826381 15391 ts_tablet_manager.cc:1428] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:58:20.939817 15396 raft_consensus.cc:482] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:20.939950 15396 raft_consensus.cc:504] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.941076 15396 leader_election.cc:288] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205), 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083)
I20260620 14:58:20.945374 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.941444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:58885 (local address 127.7.46.130:39083)
0620 14:58:20.941562 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.941566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.941735 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:58:20.942002 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.942005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.942063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.942160 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.942492 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.943270 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.944430 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.944821 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.944952 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.945017 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.945176 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.945245 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:58:20.945429 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.941375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:44293 (local address 127.7.46.129:45205)
0620 14:58:20.941526 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.941531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.941709 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:58:20.941940 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.941943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.941999 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.942084 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:20.942482 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.943270 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.944439 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.944855 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.945000 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.945052 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.945215 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.945295 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:58:20.945441 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.941369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:39083 (local address 127.7.46.131:58885)
0620 14:58:20.941652 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.941665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.941811 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.942172 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.942174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.942191 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.942349 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.942360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.943412 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.943415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.944303 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.944312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.944943 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.944946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.945045 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.945074 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.945162 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.945230 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":2,"spinlock_wait_cycles":12416,"thread_start_us":71,"threads_started":1}
I20260620 14:58:20.945449 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.941261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45205 (local address 127.7.46.131:44293)
0620 14:58:20.941596 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.941635 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.941797 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.942199 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.942202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.942237 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:20.942351 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.942364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.943379 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.943381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.944306 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.944316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.944987 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.944991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.945087 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.945113 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.945200 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.945269 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":23,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260620 14:58:20.945994 15035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4284ed90c4b00a5bf8860c4d9220d" candidate_uuid: "24ca40ce1631454d8b4a3cf5193b056c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885b8ab3500b48b49c56c974e3827a16" is_pre_election: true
I20260620 14:58:20.946079 15175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4284ed90c4b00a5bf8860c4d9220d" candidate_uuid: "24ca40ce1631454d8b4a3cf5193b056c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009a44f941f343219a27a80ea0ee23b1" is_pre_election: true
I20260620 14:58:20.946151 15035 raft_consensus.cc:2454] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ca40ce1631454d8b4a3cf5193b056c in term 0.
I20260620 14:58:20.946190 15175 raft_consensus.cc:2454] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ca40ce1631454d8b4a3cf5193b056c in term 0.
I20260620 14:58:20.946507 15246 leader_election.cc:302] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 009a44f941f343219a27a80ea0ee23b1, 24ca40ce1631454d8b4a3cf5193b056c, 885b8ab3500b48b49c56c974e3827a16; no voters:
I20260620 14:58:20.946774 15396 raft_consensus.cc:2781] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:20.946846 15396 raft_consensus.cc:482] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:20.946902 15396 raft_consensus.cc:3037] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.948012 15396 raft_consensus.cc:504] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.948333 15396 leader_election.cc:288] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [CANDIDATE]: Term 1 election: Requested vote from peers 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205), 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083)
I20260620 14:58:20.948664 15175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4284ed90c4b00a5bf8860c4d9220d" candidate_uuid: "24ca40ce1631454d8b4a3cf5193b056c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009a44f941f343219a27a80ea0ee23b1"
I20260620 14:58:20.948684 15035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4284ed90c4b00a5bf8860c4d9220d" candidate_uuid: "24ca40ce1631454d8b4a3cf5193b056c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885b8ab3500b48b49c56c974e3827a16"
I20260620 14:58:20.948783 15175 raft_consensus.cc:3037] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.948803 15035 raft_consensus.cc:3037] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:20.949707 15175 raft_consensus.cc:2454] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ca40ce1631454d8b4a3cf5193b056c in term 1.
I20260620 14:58:20.949791 15035 raft_consensus.cc:2454] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ca40ce1631454d8b4a3cf5193b056c in term 1.
I20260620 14:58:20.949981 15247 leader_election.cc:302] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009a44f941f343219a27a80ea0ee23b1, 24ca40ce1631454d8b4a3cf5193b056c; no voters:
I20260620 14:58:20.950176 15396 raft_consensus.cc:2781] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:20.950341 15396 raft_consensus.cc:686] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [term 1 LEADER]: Becoming Leader. State: Replica: 24ca40ce1631454d8b4a3cf5193b056c, State: Running, Role: LEADER
I20260620 14:58:20.950606 15396 consensus_queue.cc:237] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:20.951797 15432 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: TxnStatusTablet state changed. Reason: New leader 24ca40ce1631454d8b4a3cf5193b056c. Latest consensus state: current_term: 1 leader_uuid: "24ca40ce1631454d8b4a3cf5193b056c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.951905 15432 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:58:20.952168 15434 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:58:20.952310 15434 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:58:20.953063 14878 catalog_manager.cc:5495] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131). New cstate: current_term: 1 leader_uuid: "24ca40ce1631454d8b4a3cf5193b056c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } health_report { overall_health: HEALTHY } } }
I20260620 14:58:20.981283 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.977235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35907 (local address 127.0.0.1:44852)
0620 14:58:20.977400 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.977416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:20.977510 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:20.977835 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:20.977837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:20.977845 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:58:20.977948 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.977952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.978999 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.979006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.979613 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:20.979621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.980190 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.980193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:20.980259 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.980303 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:58:20.981028 (+ 725us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:58:20.981041 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:58:20.981099 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:58:20.981169 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:58:20.981302 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.977345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.7.46.131:35907)
0620 14:58:20.977478 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.977483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.977498 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:58:20.977650 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.977654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.977715 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.977820 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:20.978050 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.978860 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.979716 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.980086 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.980192 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.980434 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:20.980939 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:20.981050 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.981111 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.981166 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:58:20.984009 15035 raft_consensus.cc:1261] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Refusing update from remote peer 24ca40ce1631454d8b4a3cf5193b056c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:20.984161 15175 raft_consensus.cc:1261] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Refusing update from remote peer 24ca40ce1631454d8b4a3cf5193b056c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:20.984546 15432 consensus_queue.cc:1035] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [LEADER]: Connected to new peer: Peer: permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:20.984694 15396 consensus_queue.cc:1035] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c [LEADER]: Connected to new peer: Peer: permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:20.986228 15399 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: TxnStatusTablet state changed. Reason: New leader 24ca40ce1631454d8b4a3cf5193b056c. Latest consensus state: current_term: 1 leader_uuid: "24ca40ce1631454d8b4a3cf5193b056c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.986341 15399 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:58:20.986316 15397 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: TxnStatusTablet state changed. Reason: New leader 24ca40ce1631454d8b4a3cf5193b056c. Latest consensus state: current_term: 1 leader_uuid: "24ca40ce1631454d8b4a3cf5193b056c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.986436 15397 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:58:20.988041 15396 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24ca40ce1631454d8b4a3cf5193b056c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.988153 15396 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:58:20.988240 15403 mvcc.cc:205] Tried to move back new op lower bound from 7298938884025491456 to 7298938883896225792. Current Snapshot: MvccSnapshot[applied={T|T < 7298938884025491456}]
I20260620 14:58:20.988726 15399 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24ca40ce1631454d8b4a3cf5193b056c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.988828 15399 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 009a44f941f343219a27a80ea0ee23b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:58:20.990411 15432 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24ca40ce1631454d8b4a3cf5193b056c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.990559 15432 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 24ca40ce1631454d8b4a3cf5193b056c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:58:20.990835 15404 mvcc.cc:205] Tried to move back new op lower bound from 7298938884025491456 to 7298938883896225792. Current Snapshot: MvccSnapshot[applied={T|T < 7298938884025491456}]
I20260620 14:58:20.990991 15397 tablet_replica.cc:439] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24ca40ce1631454d8b4a3cf5193b056c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } } }
I20260620 14:58:20.991075 15397 tablet_replica.cc:442] T 30b4284ed90c4b00a5bf8860c4d9220d P 885b8ab3500b48b49c56c974e3827a16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:58:20.998046 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.994193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.7.46.129:45205)
0620 14:58:20.994298 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:20.994302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:20.994412 (+ 110us) server_negotiation.cc:407] Connection header received
0620 14:58:20.994558 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:20.994561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:20.994619 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:20.994742 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:20.994937 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.995711 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.996555 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:20.996901 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:20.997003 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:20.997206 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:20.997717 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:20.997826 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:20.997862 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 14:58:20.997915 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:58:21.008013 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.004639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:50911 (local address 127.7.46.131:35907)
0620 14:58:21.004770 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:21.004777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:21.004943 (+ 166us) server_negotiation.cc:407] Connection header received
0620 14:58:21.005046 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:21.005048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:21.005086 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:21.005139 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:21.005380 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.005833 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.007023 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.007222 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.007294 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.007611 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.007669 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.007719 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:21.007870 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:58:21.007913 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:58:21.008074 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.004552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35907 (local address 127.7.46.129:50911)
0620 14:58:21.004845 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:21.004860 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:21.004959 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:21.005172 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:21.005174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:21.005182 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:58:21.005284 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:21.005290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.005931 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.005934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:21.006911 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:21.006921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.007324 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.007328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:21.007425 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.007485 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.007747 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.007753 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:58:21.007838 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:58:21.007915 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20260620 14:58:21.014348 15315 raft_consensus.cc:1261] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Refusing update from remote peer 885b8ab3500b48b49c56c974e3827a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:21.014426 15175 raft_consensus.cc:1261] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Refusing update from remote peer 885b8ab3500b48b49c56c974e3827a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:21.014806 15397 consensus_queue.cc:1035] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:21.015018 15419 consensus_queue.cc:1035] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:21.037003 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.032881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:44085 (local address 127.7.46.129:45205)
0620 14:58:21.033054 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:21.033058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:21.033071 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:58:21.033111 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:21.033114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:21.033159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:21.033233 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:21.033458 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.034223 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.035219 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.035524 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.035626 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.036532 (+ 906us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.036613 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.036672 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:21.036791 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:58:21.036832 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 14:58:21.037005 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.032525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45205 (local address 127.7.46.131:44085)
0620 14:58:21.032868 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:21.032884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:21.032999 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:21.033249 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:21.033252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:21.033259 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:58:21.033356 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:21.033360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.034328 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.034330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:21.035096 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:21.035105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.036219 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.036225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:21.036322 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.036372 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.036699 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.036705 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:58:21.036774 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:58:21.036848 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":116,"threads_started":1}
I20260620 14:58:21.059496 14877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:21.059659 14877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:21.061528 14877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: Sending DeleteTablet for 3 replicas of tablet ab01a17f90a34977b57390e73577d546
I20260620 14:58:21.062181 15155 tablet_service.cc:1514] Processing DeleteTablet for tablet ab01a17f90a34977b57390e73577d546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:58094
I20260620 14:58:21.062265 15295 tablet_service.cc:1514] Processing DeleteTablet for tablet ab01a17f90a34977b57390e73577d546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:44836
I20260620 14:58:21.062276 15015 tablet_service.cc:1514] Processing DeleteTablet for tablet ab01a17f90a34977b57390e73577d546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:37026
I20260620 14:58:21.062364 15411 tablet_replica.cc:330] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: stopping tablet replica
I20260620 14:58:21.062436 15412 tablet_replica.cc:330] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: stopping tablet replica
I20260620 14:58:21.062469 15411 raft_consensus.cc:2229] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:21.062551 15412 raft_consensus.cc:2229] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:21.062573 15411 raft_consensus.cc:2258] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:21.062731 15413 tablet_replica.cc:330] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: stopping tablet replica
I20260620 14:58:21.062815 15412 raft_consensus.cc:2258] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:21.062844 15413 raft_consensus.cc:2229] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:21.062943 15413 raft_consensus.cc:2258] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:21.063562 15412 ts_tablet_manager.cc:1905] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:21.063809 15411 ts_tablet_manager.cc:1905] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:21.066578 15411 ts_tablet_manager.cc:1918] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:58:21.066661 15411 log.cc:1196] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/ab01a17f90a34977b57390e73577d546
I20260620 14:58:21.066851 15412 ts_tablet_manager.cc:1918] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:58:21.066926 15411 ts_tablet_manager.cc:1939] T ab01a17f90a34977b57390e73577d546 P 009a44f941f343219a27a80ea0ee23b1: Deleting consensus metadata
I20260620 14:58:21.066962 15412 log.cc:1196] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/ab01a17f90a34977b57390e73577d546
I20260620 14:58:21.067255 15412 ts_tablet_manager.cc:1939] T ab01a17f90a34977b57390e73577d546 P 885b8ab3500b48b49c56c974e3827a16: Deleting consensus metadata
I20260620 14:58:21.067632 14863 catalog_manager.cc:4841] TS 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083): tablet ab01a17f90a34977b57390e73577d546 (table test-table [id=d7ecfb9f978c40e7a6676d8069648a9b]) successfully deleted
I20260620 14:58:21.068073 15413 ts_tablet_manager.cc:1905] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:21.068322 14862 catalog_manager.cc:4841] TS 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205): tablet ab01a17f90a34977b57390e73577d546 (table test-table [id=d7ecfb9f978c40e7a6676d8069648a9b]) successfully deleted
I20260620 14:58:21.070649 15413 ts_tablet_manager.cc:1918] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:58:21.070731 15413 log.cc:1196] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/ab01a17f90a34977b57390e73577d546
I20260620 14:58:21.070978 15413 ts_tablet_manager.cc:1939] T ab01a17f90a34977b57390e73577d546 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting consensus metadata
I20260620 14:58:21.071633 14862 catalog_manager.cc:4841] TS 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907): tablet ab01a17f90a34977b57390e73577d546 (table test-table [id=d7ecfb9f978c40e7a6676d8069648a9b]) successfully deleted
Jun 20 14:58:21 dist-test-slave-30t4 krb5kdc[14833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:58:21 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:21 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:58:21.093219 14927 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:21.086655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.7.46.190:35921)
0620 14:58:21.086786 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:21.086790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:21.087080 (+ 290us) server_negotiation.cc:407] Connection header received
0620 14:58:21.087240 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:21.087244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:21.087299 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:21.087387 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:21.088243 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.088857 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.089770 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.089994 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.090097 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.093097 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46940: BlockingRecv error: recv got EOF from 127.0.0.1:46940 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 14:58:21 dist-test-slave-30t4 krb5kdc[14833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM
I20260620 14:58:21.103345 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.094942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.7.46.190:35921)
0620 14:58:21.095047 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:21.095051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:21.095437 (+ 386us) server_negotiation.cc:407] Connection header received
0620 14:58:21.095564 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:21.095567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:21.095606 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:21.095694 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:21.096377 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.096857 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.097637 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.097864 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.097977 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.100251 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:21.100255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:21.100257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:21.100287 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:21.101736 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:21.102160 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:21.102163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:21.102170 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:21.102240 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:21.102515 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:21.102519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:21.102521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:21.102688 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:21.102827 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:21.103069 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:58:21.103178 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:58:21.105902 14877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:21.106359 14877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:21.111310 15015 tablet_service.cc:1467] Processing CreateTablet for tablet 42575720d93a4bd48479c1e94f26fe00 (DEFAULT_TABLE table=test-table [id=6c71944518c845b1b3613cdf6a432de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:21.111399 15155 tablet_service.cc:1467] Processing CreateTablet for tablet 42575720d93a4bd48479c1e94f26fe00 (DEFAULT_TABLE table=test-table [id=6c71944518c845b1b3613cdf6a432de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:21.111615 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42575720d93a4bd48479c1e94f26fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:21.111635 15155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42575720d93a4bd48479c1e94f26fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:21.114025 15390 tablet_bootstrap.cc:492] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: Bootstrap starting.
I20260620 14:58:21.114202 15389 tablet_bootstrap.cc:492] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: Bootstrap starting.
I20260620 14:58:21.114962 15390 tablet_bootstrap.cc:654] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:21.114962 15389 tablet_bootstrap.cc:654] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:21.116000 15295 tablet_service.cc:1467] Processing CreateTablet for tablet 42575720d93a4bd48479c1e94f26fe00 (DEFAULT_TABLE table=test-table [id=6c71944518c845b1b3613cdf6a432de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:21.116226 15295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42575720d93a4bd48479c1e94f26fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:21.117947 15391 tablet_bootstrap.cc:492] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: Bootstrap starting.
I20260620 14:58:21.118875 15391 tablet_bootstrap.cc:654] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:21.119148 15390 tablet_bootstrap.cc:492] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: No bootstrap required, opened a new log
I20260620 14:58:21.119233 15390 ts_tablet_manager.cc:1397] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 14:58:21.119679 15390 raft_consensus.cc:348] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.119833 15390 raft_consensus.cc:374] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:21.119875 15390 raft_consensus.cc:729] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885b8ab3500b48b49c56c974e3827a16, State: Initialized, Role: FOLLOWER
I20260620 14:58:21.120013 15390 consensus_queue.cc:260] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.120352 15390 ts_tablet_manager.cc:1428] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:21.120755 15389 tablet_bootstrap.cc:492] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: No bootstrap required, opened a new log
I20260620 14:58:21.120837 15389 ts_tablet_manager.cc:1397] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 14:58:21.121271 15389 raft_consensus.cc:348] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.121379 15389 raft_consensus.cc:374] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:21.121410 15389 raft_consensus.cc:729] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009a44f941f343219a27a80ea0ee23b1, State: Initialized, Role: FOLLOWER
I20260620 14:58:21.121519 15389 consensus_queue.cc:260] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.121763 15389 ts_tablet_manager.cc:1428] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:21.122655 15391 tablet_bootstrap.cc:492] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: No bootstrap required, opened a new log
I20260620 14:58:21.122736 15391 ts_tablet_manager.cc:1397] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 14:58:21.123621 15391 raft_consensus.cc:348] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.123728 15391 raft_consensus.cc:374] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:21.123759 15391 raft_consensus.cc:729] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ca40ce1631454d8b4a3cf5193b056c, State: Initialized, Role: FOLLOWER
I20260620 14:58:21.123889 15391 consensus_queue.cc:260] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.124167 15391 ts_tablet_manager.cc:1428] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:58:21.257208 15445 raft_consensus.cc:482] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:21.257340 15445 raft_consensus.cc:504] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.257828 15445 leader_election.cc:288] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083), 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:21.258257 15175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42575720d93a4bd48479c1e94f26fe00" candidate_uuid: "885b8ab3500b48b49c56c974e3827a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009a44f941f343219a27a80ea0ee23b1" is_pre_election: true
I20260620 14:58:21.258319 15315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42575720d93a4bd48479c1e94f26fe00" candidate_uuid: "885b8ab3500b48b49c56c974e3827a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca40ce1631454d8b4a3cf5193b056c" is_pre_election: true
I20260620 14:58:21.258440 15175 raft_consensus.cc:2454] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 885b8ab3500b48b49c56c974e3827a16 in term 0.
I20260620 14:58:21.258481 15315 raft_consensus.cc:2454] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 885b8ab3500b48b49c56c974e3827a16 in term 0.
I20260620 14:58:21.258729 14967 leader_election.cc:302] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009a44f941f343219a27a80ea0ee23b1, 885b8ab3500b48b49c56c974e3827a16; no voters:
I20260620 14:58:21.258891 15445 raft_consensus.cc:2781] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:21.258950 15445 raft_consensus.cc:482] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:21.258980 15445 raft_consensus.cc:3037] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:21.260021 15445 raft_consensus.cc:504] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.260370 15445 leader_election.cc:288] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [CANDIDATE]: Term 1 election: Requested vote from peers 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083), 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:21.260696 15175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42575720d93a4bd48479c1e94f26fe00" candidate_uuid: "885b8ab3500b48b49c56c974e3827a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009a44f941f343219a27a80ea0ee23b1"
I20260620 14:58:21.260763 15315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42575720d93a4bd48479c1e94f26fe00" candidate_uuid: "885b8ab3500b48b49c56c974e3827a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca40ce1631454d8b4a3cf5193b056c"
I20260620 14:58:21.260815 15175 raft_consensus.cc:3037] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:21.260902 15315 raft_consensus.cc:3037] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:21.261729 15175 raft_consensus.cc:2454] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885b8ab3500b48b49c56c974e3827a16 in term 1.
I20260620 14:58:21.261817 15315 raft_consensus.cc:2454] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885b8ab3500b48b49c56c974e3827a16 in term 1.
I20260620 14:58:21.262099 14967 leader_election.cc:302] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009a44f941f343219a27a80ea0ee23b1, 885b8ab3500b48b49c56c974e3827a16; no voters:
I20260620 14:58:21.262274 15445 raft_consensus.cc:2781] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:21.262389 15445 raft_consensus.cc:686] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 1 LEADER]: Becoming Leader. State: Replica: 885b8ab3500b48b49c56c974e3827a16, State: Running, Role: LEADER
I20260620 14:58:21.262533 15445 consensus_queue.cc:237] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.264202 14878 catalog_manager.cc:5495] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129). New cstate: current_term: 1 leader_uuid: "885b8ab3500b48b49c56c974e3827a16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } health_report { overall_health: UNKNOWN } } }
I20260620 14:58:21.280603 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.276965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.7.46.129:45205)
0620 14:58:21.277092 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:21.277099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:21.277117 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:58:21.277274 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:21.277277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:21.277328 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:21.277410 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:21.277675 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.278169 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.279314 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.279713 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.279857 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.280163 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.280247 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.280326 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:21.280421 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:58:21.280467 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:21.282518 15175 raft_consensus.cc:1261] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Refusing update from remote peer 885b8ab3500b48b49c56c974e3827a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:21.282519 15315 raft_consensus.cc:1261] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Refusing update from remote peer 885b8ab3500b48b49c56c974e3827a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:21.282905 15445 consensus_queue.cc:1035] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:21.283077 15419 consensus_queue.cc:1035] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:21.290180 14878 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:21.290293 14878 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:21.292078 14878 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: Sending DeleteTablet for 3 replicas of tablet 42575720d93a4bd48479c1e94f26fe00
I20260620 14:58:21.292596 15015 tablet_service.cc:1514] Processing DeleteTablet for tablet 42575720d93a4bd48479c1e94f26fe00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:37026
I20260620 14:58:21.292596 15155 tablet_service.cc:1514] Processing DeleteTablet for tablet 42575720d93a4bd48479c1e94f26fe00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:58094
I20260620 14:58:21.292764 15295 tablet_service.cc:1514] Processing DeleteTablet for tablet 42575720d93a4bd48479c1e94f26fe00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:44836
I20260620 14:58:21.292786 15411 tablet_replica.cc:330] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: stopping tablet replica
I20260620 14:58:21.292913 15411 raft_consensus.cc:2229] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:21.292955 15412 tablet_replica.cc:330] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: stopping tablet replica
I20260620 14:58:21.293061 15411 raft_consensus.cc:2258] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:21.293064 15412 raft_consensus.cc:2229] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:21.293217 15413 tablet_replica.cc:330] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: stopping tablet replica
I20260620 14:58:21.293254 15412 raft_consensus.cc:2258] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:21.293309 15413 raft_consensus.cc:2229] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:21.293286 14878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:58:21.293427 15413 raft_consensus.cc:2258] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:58:21.293731 14878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:21.293972 15412 ts_tablet_manager.cc:1905] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:21.294061 15413 ts_tablet_manager.cc:1905] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:21.294066 15411 ts_tablet_manager.cc:1905] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:21.296180 15412 ts_tablet_manager.cc:1918] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:21.296233 15412 log.cc:1196] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/42575720d93a4bd48479c1e94f26fe00
I20260620 14:58:21.296329 15411 ts_tablet_manager.cc:1918] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:21.296329 15413 ts_tablet_manager.cc:1918] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:21.296398 15413 log.cc:1196] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/42575720d93a4bd48479c1e94f26fe00
I20260620 14:58:21.296398 15411 log.cc:1196] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/42575720d93a4bd48479c1e94f26fe00
I20260620 14:58:21.296416 15412 ts_tablet_manager.cc:1939] T 42575720d93a4bd48479c1e94f26fe00 P 885b8ab3500b48b49c56c974e3827a16: Deleting consensus metadata
I20260620 14:58:21.296651 15413 ts_tablet_manager.cc:1939] T 42575720d93a4bd48479c1e94f26fe00 P 24ca40ce1631454d8b4a3cf5193b056c: Deleting consensus metadata
I20260620 14:58:21.296651 15411 ts_tablet_manager.cc:1939] T 42575720d93a4bd48479c1e94f26fe00 P 009a44f941f343219a27a80ea0ee23b1: Deleting consensus metadata
I20260620 14:58:21.297101 14862 catalog_manager.cc:4841] TS 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205): tablet 42575720d93a4bd48479c1e94f26fe00 (table test-table [id=6c71944518c845b1b3613cdf6a432de6]) successfully deleted
I20260620 14:58:21.297233 14863 catalog_manager.cc:4841] TS 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083): tablet 42575720d93a4bd48479c1e94f26fe00 (table test-table [id=6c71944518c845b1b3613cdf6a432de6]) successfully deleted
I20260620 14:58:21.297307 14862 catalog_manager.cc:4841] TS 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907): tablet 42575720d93a4bd48479c1e94f26fe00 (table test-table [id=6c71944518c845b1b3613cdf6a432de6]) successfully deleted
I20260620 14:58:21.300040 15155 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5dc0cc057d49b2854ec4ade4687cff (DEFAULT_TABLE table=test-table [id=aaa3de2950db49069dad69abf3912e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:21.300043 15015 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5dc0cc057d49b2854ec4ade4687cff (DEFAULT_TABLE table=test-table [id=aaa3de2950db49069dad69abf3912e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:21.300302 15155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5dc0cc057d49b2854ec4ade4687cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:21.300300 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5dc0cc057d49b2854ec4ade4687cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:21.301847 15390 tablet_bootstrap.cc:492] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16: Bootstrap starting.
I20260620 14:58:21.301869 15389 tablet_bootstrap.cc:492] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: Bootstrap starting.
I20260620 14:58:21.302038 15295 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5dc0cc057d49b2854ec4ade4687cff (DEFAULT_TABLE table=test-table [id=aaa3de2950db49069dad69abf3912e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:21.302259 15295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5dc0cc057d49b2854ec4ade4687cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:21.302620 15389 tablet_bootstrap.cc:654] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:21.302663 15390 tablet_bootstrap.cc:654] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:21.303580 15389 tablet_bootstrap.cc:492] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: No bootstrap required, opened a new log
I20260620 14:58:21.303678 15389 ts_tablet_manager.cc:1397] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:21.304131 15389 raft_consensus.cc:348] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.304263 15389 raft_consensus.cc:374] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:21.304308 15389 raft_consensus.cc:729] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009a44f941f343219a27a80ea0ee23b1, State: Initialized, Role: FOLLOWER
I20260620 14:58:21.304452 15391 tablet_bootstrap.cc:492] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: Bootstrap starting.
I20260620 14:58:21.304448 15389 consensus_queue.cc:260] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.304703 15389 ts_tablet_manager.cc:1428] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:21.304791 15390 tablet_bootstrap.cc:492] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16: No bootstrap required, opened a new log
I20260620 14:58:21.304872 15390 ts_tablet_manager.cc:1397] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:58:21.305226 15391 tablet_bootstrap.cc:654] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:21.305279 15390 raft_consensus.cc:348] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.305387 15390 raft_consensus.cc:374] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:21.305438 15390 raft_consensus.cc:729] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885b8ab3500b48b49c56c974e3827a16, State: Initialized, Role: FOLLOWER
I20260620 14:58:21.305552 15390 consensus_queue.cc:260] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.305812 15390 ts_tablet_manager.cc:1428] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:58:21.306712 15391 tablet_bootstrap.cc:492] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: No bootstrap required, opened a new log
I20260620 14:58:21.306795 15391 ts_tablet_manager.cc:1397] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:58:21.307194 15391 raft_consensus.cc:348] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.307278 15391 raft_consensus.cc:374] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:21.307299 15391 raft_consensus.cc:729] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ca40ce1631454d8b4a3cf5193b056c, State: Initialized, Role: FOLLOWER
I20260620 14:58:21.307444 15391 consensus_queue.cc:260] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.307714 15391 ts_tablet_manager.cc:1428] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:58:21.731012 15399 raft_consensus.cc:482] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:21.731174 15399 raft_consensus.cc:504] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.731593 15399 leader_election.cc:288] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205), 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:21.732163 15035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5dc0cc057d49b2854ec4ade4687cff" candidate_uuid: "009a44f941f343219a27a80ea0ee23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885b8ab3500b48b49c56c974e3827a16" is_pre_election: true
I20260620 14:58:21.732220 15315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5dc0cc057d49b2854ec4ade4687cff" candidate_uuid: "009a44f941f343219a27a80ea0ee23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca40ce1631454d8b4a3cf5193b056c" is_pre_election: true
I20260620 14:58:21.732323 15035 raft_consensus.cc:2454] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009a44f941f343219a27a80ea0ee23b1 in term 0.
I20260620 14:58:21.732405 15315 raft_consensus.cc:2454] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009a44f941f343219a27a80ea0ee23b1 in term 0.
I20260620 14:58:21.732697 15106 leader_election.cc:302] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009a44f941f343219a27a80ea0ee23b1, 885b8ab3500b48b49c56c974e3827a16; no voters:
I20260620 14:58:21.732908 15399 raft_consensus.cc:2781] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:21.732986 15399 raft_consensus.cc:482] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:21.733040 15399 raft_consensus.cc:3037] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:21.734036 15399 raft_consensus.cc:504] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.734359 15399 leader_election.cc:288] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [CANDIDATE]: Term 1 election: Requested vote from peers 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205), 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907)
I20260620 14:58:21.734745 15035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5dc0cc057d49b2854ec4ade4687cff" candidate_uuid: "009a44f941f343219a27a80ea0ee23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885b8ab3500b48b49c56c974e3827a16"
I20260620 14:58:21.734793 15315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5dc0cc057d49b2854ec4ade4687cff" candidate_uuid: "009a44f941f343219a27a80ea0ee23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca40ce1631454d8b4a3cf5193b056c"
I20260620 14:58:21.734884 15035 raft_consensus.cc:3037] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:21.734911 15315 raft_consensus.cc:3037] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:21.735999 15315 raft_consensus.cc:2454] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009a44f941f343219a27a80ea0ee23b1 in term 1.
I20260620 14:58:21.736023 15035 raft_consensus.cc:2454] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009a44f941f343219a27a80ea0ee23b1 in term 1.
I20260620 14:58:21.736275 15106 leader_election.cc:302] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009a44f941f343219a27a80ea0ee23b1, 24ca40ce1631454d8b4a3cf5193b056c; no voters:
I20260620 14:58:21.736454 15399 raft_consensus.cc:2781] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:21.736644 15399 raft_consensus.cc:686] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 1 LEADER]: Becoming Leader. State: Replica: 009a44f941f343219a27a80ea0ee23b1, State: Running, Role: LEADER
I20260620 14:58:21.736807 15399 consensus_queue.cc:237] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } }
I20260620 14:58:21.738498 14877 catalog_manager.cc:5495] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130). New cstate: current_term: 1 leader_uuid: "009a44f941f343219a27a80ea0ee23b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009a44f941f343219a27a80ea0ee23b1" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 39083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 } health_report { overall_health: UNKNOWN } } }
I20260620 14:58:21.762048 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.759261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.7.46.130:39083)
0620 14:58:21.759453 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:21.759456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:21.759467 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:58:21.759505 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:21.759507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:21.759545 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:21.759613 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:21.759836 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.760337 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.761097 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.761305 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.761375 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.761604 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.761666 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.761716 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:21.761797 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:58:21.761837 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260620 14:58:21.771293 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.767985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35907 (local address 127.7.46.130:42799)
0620 14:58:21.768324 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:21.768338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:21.768438 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:21.768679 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:21.768682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:21.768690 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:58:21.768781 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:21.768786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.769526 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.769530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:21.770128 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:21.770136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.770600 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.770604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:21.770686 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.770733 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.771029 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.771037 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:58:21.771100 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:58:21.771171 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260620 14:58:21.771294 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.768100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:42799 (local address 127.7.46.131:35907)
0620 14:58:21.768298 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:21.768303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:21.768412 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:58:21.768525 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:21.768528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:21.768577 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:21.768668 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:21.768893 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.769404 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.770250 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.770482 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.770599 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.770867 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.770941 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.771022 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:21.771108 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:58:21.771154 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
I20260620 14:58:21.776636 15315 raft_consensus.cc:1261] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Refusing update from remote peer 009a44f941f343219a27a80ea0ee23b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:21.776655 15035 raft_consensus.cc:1261] T 5e5dc0cc057d49b2854ec4ade4687cff P 885b8ab3500b48b49c56c974e3827a16 [term 1 FOLLOWER]: Refusing update from remote peer 009a44f941f343219a27a80ea0ee23b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:21.777014 15472 consensus_queue.cc:1035] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ca40ce1631454d8b4a3cf5193b056c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:21.777168 15399 consensus_queue.cc:1035] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "885b8ab3500b48b49c56c974e3827a16" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:21.793808 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.790597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:45983 (local address 127.7.46.130:39083)
0620 14:58:21.790718 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:21.790723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:21.790865 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:58:21.791016 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:21.791019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:21.791067 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:21.791144 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:21.791400 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.792054 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.792844 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.793048 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.793154 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.793370 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.793444 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.793498 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:21.793631 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:58:21.793685 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:58:21.793836 15485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.790537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:39083 (local address 127.7.46.131:45983)
0620 14:58:21.790788 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:21.790802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:21.790905 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:21.791164 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:21.791166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:21.791175 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:58:21.791281 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:21.791287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.792151 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.792154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:21.792747 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:21.792755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:21.793136 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:21.793138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:21.793199 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:21.793246 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:58:21.793517 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:58:21.793524 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:58:21.793615 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:58:21.793701 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
I20260620 14:58:21.813576 14877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:21.813709 14877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:21.815521 14877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc061f65acee41fc96f0c770b44422bd: Sending DeleteTablet for 3 replicas of tablet 5e5dc0cc057d49b2854ec4ade4687cff
I20260620 14:58:21.816047 15295 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5dc0cc057d49b2854ec4ade4687cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:44836
I20260620 14:58:21.816045 15015 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5dc0cc057d49b2854ec4ade4687cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:37026
I20260620 14:58:21.816051 15155 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5dc0cc057d49b2854ec4ade4687cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:21 UTC) from {username='oryx', principal='oryx/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:58094
I20260620 14:58:21.816318 15490 tablet_replica.cc:330] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: stopping tablet replica
I20260620 14:58:21.816438 15490 raft_consensus.cc:2229] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:21.816641 15490 raft_consensus.cc:2258] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:21.816658 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 14950
I20260620 14:58:21.817903 15488 tablet_replica.cc:330] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: stopping tablet replica
I20260620 14:58:21.818032 15488 raft_consensus.cc:2229] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:21.818131 15488 raft_consensus.cc:2258] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:21.818220 15490 ts_tablet_manager.cc:1905] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:21.820145 15488 ts_tablet_manager.cc:1905] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:21.821564 15490 ts_tablet_manager.cc:1918] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:58:21.821650 15490 log.cc:1196] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/5e5dc0cc057d49b2854ec4ade4687cff
I20260620 14:58:21.821938 15490 ts_tablet_manager.cc:1939] T 5e5dc0cc057d49b2854ec4ade4687cff P 009a44f941f343219a27a80ea0ee23b1: Deleting consensus metadata
I20260620 14:58:21.822283 15488 ts_tablet_manager.cc:1918] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:58:21.822376 15488 log.cc:1196] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/5e5dc0cc057d49b2854ec4ade4687cff
I20260620 14:58:21.822685 15488 ts_tablet_manager.cc:1939] T 5e5dc0cc057d49b2854ec4ade4687cff P 24ca40ce1631454d8b4a3cf5193b056c: Deleting consensus metadata
I20260620 14:58:21.822675 14863 catalog_manager.cc:4841] TS 009a44f941f343219a27a80ea0ee23b1 (127.7.46.130:39083): tablet 5e5dc0cc057d49b2854ec4ade4687cff (table test-table [id=aaa3de2950db49069dad69abf3912e42]) successfully deleted
I20260620 14:58:21.823248 14862 catalog_manager.cc:4841] TS 24ca40ce1631454d8b4a3cf5193b056c (127.7.46.131:35907): tablet 5e5dc0cc057d49b2854ec4ade4687cff (table test-table [id=aaa3de2950db49069dad69abf3912e42]) successfully deleted
W20260620 14:58:21.825579 14862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:45205 (error 108)
I20260620 14:58:21.825801 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 15090
I20260620 14:58:21.827232 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.826702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45205 (local address 127.0.0.1:37218)
0620 14:58:21.827019 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:21.827105 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":53,"threads_started":1}
W20260620 14:58:21.827389 14862 catalog_manager.cc:4584] TS 885b8ab3500b48b49c56c974e3827a16 (127.7.46.129:45205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e5dc0cc057d49b2854ec4ade4687cff: Network error: Client connection negotiation failed: client connection to 127.7.46.129:45205: connect: Connection refused (error 111)
I20260620 14:58:21.832975 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 15230
I20260620 14:58:21.839417 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 14846
2026-06-20T14:58:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3572 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:21 dist-test-slave-30t4 krb5kdc[15495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:21 dist-test-slave-30t4 krb5kdc[15495](info): set up 2 sockets
Jun 20 14:58:21 dist-test-slave-30t4 krb5kdc[15495](info): commencing operation
krb5kdc: starting...
W20260620 14:58:22.880508 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:22 dist-test-slave-30t4 krb5kdc[15495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:58:22.998657 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:36509
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:36509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:23.105511 15508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:23.105777 15508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:23.105856 15508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:23.109720 15508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:23.109805 15508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:23.109843 15508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:23.109877 15508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:23.109910 15508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:23.114953 15508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:36509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:36509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:23.115919 15508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:23.116866 15508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:23.125108 15516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:23.125170 15513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:23.125355 15508 server_base.cc:1034] running on GCE node
W20260620 14:58:23.125170 15514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:23.125901 15508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:23.126955 15508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:23.128113 15508 hybrid_clock.cc:648] HybridClock initialized: now 1781967503128081 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:58:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1296 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:23 dist-test-slave-30t4 krb5kdc[15521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:23 dist-test-slave-30t4 krb5kdc[15521](info): set up 2 sockets
Jun 20 14:58:23 dist-test-slave-30t4 krb5kdc[15521](info): commencing operation
krb5kdc: starting...
W20260620 14:58:24.180444 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:24 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:58:24.296897 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:40813
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:24.404296 15534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:24.404556 15534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:24.404596 15534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:24.408571 15534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:24.408622 15534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:24.408641 15534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:24.408658 15534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:24.408672 15534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:24.413992 15534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:40813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:24.415131 15534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:24.416359 15534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:24.424652 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:24.424774 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:24.424656 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:24.424871 15534 server_base.cc:1034] running on GCE node
I20260620 14:58:24.425542 15534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:24.426609 15534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:24.427824 15534 hybrid_clock.cc:648] HybridClock initialized: now 1781967504427798 us; error 29 us; skew 500 ppm
Jun 20 14:58:24 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967504, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:24.430372 15534 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:58:24.431155 15534 webserver.cc:460] Webserver started at http://127.7.46.190:43509/ using document root <none> and password file <none>
I20260620 14:58:24.431649 15534 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:24.431705 15534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:24.431969 15534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:24.433876 15534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "792ff56996db4c068552e41bdd91d4bc"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "bea3e3273416e2ce06fa1acbafb1207a"
server_key_iv: "1ad14e21694091c8a3b4af1fb596837e"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.434379 15534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "792ff56996db4c068552e41bdd91d4bc"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "bea3e3273416e2ce06fa1acbafb1207a"
server_key_iv: "1ad14e21694091c8a3b4af1fb596837e"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.438082 15534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:24.440408 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:24.441606 15534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:24.441705 15534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "792ff56996db4c068552e41bdd91d4bc"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "bea3e3273416e2ce06fa1acbafb1207a"
server_key_iv: "1ad14e21694091c8a3b4af1fb596837e"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.441787 15534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:24.456182 15534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:24.459384 15534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:24.459537 15534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:24.466868 15534 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:40813
I20260620 14:58:24.466892 15601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:40813 every 8 connection(s)
I20260620 14:58:24.467839 15534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:24.470464 15602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:24.472610 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 15534
I20260620 14:58:24.472714 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:24.473034 7354 external_mini_cluster.cc:1428] Setting key 9489c90d1e3cc8e42cd030e1859b0a50
I20260620 14:58:24.476349 15602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Bootstrap starting.
I20260620 14:58:24.478440 15602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:24.479158 15602 log.cc:824] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:24.480924 15602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: No bootstrap required, opened a new log
Jun 20 14:58:24 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:24.483728 15602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } }
I20260620 14:58:24.484032 15602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:24.484118 15602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792ff56996db4c068552e41bdd91d4bc, State: Initialized, Role: FOLLOWER
I20260620 14:58:24.484547 15602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } }
I20260620 14:58:24.484697 15602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:24.484786 15602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:24.484886 15602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:24.486021 15602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } }
I20260620 14:58:24.486399 15602 leader_election.cc:302] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792ff56996db4c068552e41bdd91d4bc; no voters:
I20260620 14:58:24.486742 15602 leader_election.cc:288] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:24.486951 15607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:24.487269 15607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 1 LEADER]: Becoming Leader. State: Replica: 792ff56996db4c068552e41bdd91d4bc, State: Running, Role: LEADER
I20260620 14:58:24.487687 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:24.474405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.7.46.190:40813)
0620 14:58:24.474829 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:24.474844 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:24.474878 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:58:24.475493 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:24.475509 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:24.475896 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:24.476226 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:24.477022 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.477979 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.478881 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.479399 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.479574 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:24.481999 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:24.482010 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:24.482023 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:24.482049 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:24.484483 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:24.484961 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:24.484968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:24.484975 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:24.485120 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:24.485407 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:24.485410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:24.485412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:24.485929 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:24.486132 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:24.486414 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:58:24.486712 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260620 14:58:24.487994 15602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:24.487672 15607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } }
I20260620 14:58:24.489250 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792ff56996db4c068552e41bdd91d4bc. Latest consensus state: current_term: 1 leader_uuid: "792ff56996db4c068552e41bdd91d4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } } }
I20260620 14:58:24.489467 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:24.489710 15608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "792ff56996db4c068552e41bdd91d4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } } }
I20260620 14:58:24.489825 15608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:24.490150 15620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:24.493304 15620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:24.498775 15620 catalog_manager.cc:1347] Generated new cluster ID: f0fc293d740d49aabcee7d582ca38615
I20260620 14:58:24.498847 15620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:24.515731 15620 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:24.516669 15620 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:24.523469 15620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Generated new TSK 0
I20260620 14:58:24.524170 15620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:58:24.558460 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40813
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:24.663200 15628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:24.663476 15628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:24.663568 15628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:24.667428 15628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:24.667510 15628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:24.667598 15628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:24.672547 15628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:24.673476 15628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:24.674414 15628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:24.683506 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:24.683473 15634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:24.683597 15636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:24.683746 15628 server_base.cc:1034] running on GCE node
I20260620 14:58:24.684197 15628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:24.684811 15628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:24.685987 15628 hybrid_clock.cc:648] HybridClock initialized: now 1781967504685977 us; error 34 us; skew 500 ppm
Jun 20 14:58:24 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967504, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:24.688808 15628 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:58:24.689572 15628 webserver.cc:460] Webserver started at http://127.7.46.129:38483/ using document root <none> and password file <none>
I20260620 14:58:24.690116 15628 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:24.690174 15628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:24.690323 15628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:24.692181 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "f0e13927c6b343feba1777075056fe6a"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "8f8e15ef0b5770eb301f8e141c46480d"
server_key_iv: "b1829932ec2f9958e7b4c09ed303a2c9"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.692924 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "f0e13927c6b343feba1777075056fe6a"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "8f8e15ef0b5770eb301f8e141c46480d"
server_key_iv: "b1829932ec2f9958e7b4c09ed303a2c9"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.696561 15628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:24.698918 15643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:24.700186 15628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:24.700274 15628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "f0e13927c6b343feba1777075056fe6a"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "8f8e15ef0b5770eb301f8e141c46480d"
server_key_iv: "b1829932ec2f9958e7b4c09ed303a2c9"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.700352 15628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:24.723524 15628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:24.726634 15628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:24.726773 15628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:24.727250 15628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:24.727968 15628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:24.728012 15628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:24.728049 15628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:24.728071 15628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:24.736497 15628 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:43673
I20260620 14:58:24.736528 15756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:43673 every 8 connection(s)
I20260620 14:58:24.737347 15628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:24.744663 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 15628
I20260620 14:58:24.744814 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:24.745231 7354 external_mini_cluster.cc:1428] Setting key a5a43fc5217d5ac11a35a43e366c6227
Jun 20 14:58:24 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967504, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:24.752187 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:24.738929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:38951 (local address 127.7.46.190:40813)
0620 14:58:24.739068 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:24.739073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:24.739945 (+ 872us) server_negotiation.cc:407] Connection header received
0620 14:58:24.741142 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:24.741144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:24.741187 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:24.741266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:24.742454 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.742963 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.744062 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.744272 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.744348 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:24.747333 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:24.747338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:24.747341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:24.747372 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:24.749347 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:24.750019 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:24.750023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:24.750025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:24.750132 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:24.750898 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:24.750903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:24.750906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:24.751135 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:24.751214 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:24.751898 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 14:58:24.752022 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:24.753058 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:24.739324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:38951)
0620 14:58:24.739739 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:24.739855 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:24.740958 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:24.741475 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:24.741481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:24.741854 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:24.742316 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:24.742330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.743089 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.743091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:24.743963 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:24.743972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.744427 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.744434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:24.744634 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:24.745272 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:24.745312 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:24.747134 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:24.749499 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:24.749506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:24.749523 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:24.749885 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:24.750580 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:24.750585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:24.750589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:24.750771 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:24.751310 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:24.751319 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:24.751621 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 14:58:24.751855 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 14:58:24.752156 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":113,"threads_started":1}
I20260620 14:58:24.754920 15757 heartbeater.cc:347] Connected to a master server at 127.7.46.190:40813
I20260620 14:58:24.755180 15757 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:24.755852 15757 heartbeater.cc:511] Master 127.7.46.190:40813 requested a full tablet report, sending...
I20260620 14:58:24.757349 15566 ts_manager.cc:194] Registered new tserver with Master: f0e13927c6b343feba1777075056fe6a (127.7.46.129:43673)
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 14:58:24.758489 15566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:38951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:58:24.774670 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40813
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:24.883215 15762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:24.883432 15762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:24.883499 15762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:24.887431 15762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:24.887483 15762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:24.887598 15762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:24.892943 15762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:24.893867 15762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:24.894855 15762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:24.904121 15767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:24.904121 15768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:24.904145 15770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:24.904213 15762 server_base.cc:1034] running on GCE node
I20260620 14:58:24.904747 15762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:24.905377 15762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:24.906566 15762 hybrid_clock.cc:648] HybridClock initialized: now 1781967504906539 us; error 48 us; skew 500 ppm
Jun 20 14:58:24 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967504, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:24.909132 15762 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:58:24.909973 15762 webserver.cc:460] Webserver started at http://127.7.46.130:44081/ using document root <none> and password file <none>
I20260620 14:58:24.910503 15762 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:24.910594 15762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:24.910838 15762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:24.913115 15762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "26816681acab47d1b669201c0aca7748"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "50df5e1820034db6d85f0c7e99ffbf08"
server_key_iv: "4be425d31c6691e82dbae15c64ab8067"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.913705 15762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "26816681acab47d1b669201c0aca7748"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "50df5e1820034db6d85f0c7e99ffbf08"
server_key_iv: "4be425d31c6691e82dbae15c64ab8067"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.917397 15762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 14:58:24.919917 15777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:24.921191 15762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:58:24.921298 15762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "26816681acab47d1b669201c0aca7748"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "50df5e1820034db6d85f0c7e99ffbf08"
server_key_iv: "4be425d31c6691e82dbae15c64ab8067"
server_key_version: "encryptionkey@0"
I20260620 14:58:24.921381 15762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:24.941962 15762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:24.945298 15762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:24.945488 15762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:24.946051 15762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:24.946894 15762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:24.946949 15762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:24.947018 15762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:24.947054 15762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:24.955629 15762 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:42103
I20260620 14:58:24.955658 15890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:42103 every 8 connection(s)
I20260620 14:58:24.956506 15762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:24.960282 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 15762
I20260620 14:58:24.960379 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:24.960724 7354 external_mini_cluster.cc:1428] Setting key 7af574320a29679cf2752654b3d59522
Jun 20 14:58:24 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967504, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
I20260620 14:58:24.971381 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:24.958202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:44557 (local address 127.7.46.190:40813)
0620 14:58:24.958357 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:24.958362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:24.959230 (+ 868us) server_negotiation.cc:407] Connection header received
0620 14:58:24.960666 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:24.960671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:24.960727 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:24.960827 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:24.962327 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.963102 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.964654 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.964863 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.964949 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:24.967622 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:24.967629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:24.967634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:24.967667 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:24.969353 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:24.969845 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:24.969851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:24.969853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:24.969935 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:24.970249 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:24.970256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:24.970259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:24.970521 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:24.970630 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:24.971105 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 14:58:24.971239 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:58:24.972226 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:24.958582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:44557)
0620 14:58:24.959085 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:24.959126 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:24.960456 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:24.961015 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:24.961022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:24.961521 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:24.962176 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:24.962196 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.963221 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.963223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:24.964519 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:24.964532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:24.965014 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:24.965021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:24.965243 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:24.965846 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:24.965872 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:24.967452 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:24.969479 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:24.969484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:24.969495 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:24.969729 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:24.970034 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:24.970036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:24.970037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:24.970153 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:24.970625 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:24.970630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:24.970839 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 14:58:24.971043 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 14:58:24.971286 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":109,"threads_started":1}
I20260620 14:58:24.973431 15891 heartbeater.cc:347] Connected to a master server at 127.7.46.190:40813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
I20260620 14:58:24.973665 15891 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:24.974233 15891 heartbeater.cc:511] Master 127.7.46.190:40813 requested a full tablet report, sending...
I20260620 14:58:24.975307 15566 ts_manager.cc:194] Registered new tserver with Master: 26816681acab47d1b669201c0aca7748 (127.7.46.130:42103)
I20260620 14:58:24.975934 15566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:44557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:58:24.988891 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40813
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:25.095503 15896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:25.095815 15896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:25.095933 15896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:25.099929 15896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:25.100005 15896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:25.100095 15896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:25.105841 15896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:25.106729 15896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:25.107626 15896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:25.116791 15904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.116787 15901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.116870 15902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:25.116993 15896 server_base.cc:1034] running on GCE node
I20260620 14:58:25.117435 15896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:25.118105 15896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:25.119280 15896 hybrid_clock.cc:648] HybridClock initialized: now 1781967505119261 us; error 29 us; skew 500 ppm
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967505, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:25.121978 15896 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:58:25.122785 15896 webserver.cc:460] Webserver started at http://127.7.46.131:35093/ using document root <none> and password file <none>
I20260620 14:58:25.123318 15896 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:25.123446 15896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:25.123674 15896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:25.125682 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "346602ec968b4dc89b65a65cca25cfcb"
format_stamp: "Formatted at 2026-06-20 14:58:25 on dist-test-slave-30t4"
server_key: "d1a77c518c3730735f8f1d47331d5b65"
server_key_iv: "181f6b89aafd272d20effed4acd5067b"
server_key_version: "encryptionkey@0"
I20260620 14:58:25.126219 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "346602ec968b4dc89b65a65cca25cfcb"
format_stamp: "Formatted at 2026-06-20 14:58:25 on dist-test-slave-30t4"
server_key: "d1a77c518c3730735f8f1d47331d5b65"
server_key_iv: "181f6b89aafd272d20effed4acd5067b"
server_key_version: "encryptionkey@0"
I20260620 14:58:25.129724 15896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:25.131979 15911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.133096 15896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:25.133185 15896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "346602ec968b4dc89b65a65cca25cfcb"
format_stamp: "Formatted at 2026-06-20 14:58:25 on dist-test-slave-30t4"
server_key: "d1a77c518c3730735f8f1d47331d5b65"
server_key_iv: "181f6b89aafd272d20effed4acd5067b"
server_key_version: "encryptionkey@0"
I20260620 14:58:25.133308 15896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:25.149300 15896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:25.152215 15896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:25.152421 15896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:25.152993 15896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:25.153787 15896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:25.153831 15896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.153905 15896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:25.153944 15896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.163252 15896 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:45819
I20260620 14:58:25.163272 16024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:45819 every 8 connection(s)
I20260620 14:58:25.164181 15896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:25.164500 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 15896
I20260620 14:58:25.164638 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:25.164995 7354 external_mini_cluster.cc:1428] Setting key fb8d567ba61d1a5975a5376d1937714f
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967505, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:25.177081 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.167660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:58865 (local address 127.7.46.190:40813)
0620 14:58:25.167900 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.167904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.167918 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:25.168643 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.168646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.168688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.168763 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.169905 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.170427 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.171484 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.171687 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.171844 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.174092 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:25.174095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:25.174098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:25.174120 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:25.175379 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:25.175868 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:25.175870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:25.175871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:25.175952 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:25.176232 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:25.176235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:25.176236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:25.176389 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:25.176464 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:25.176862 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:58:25.176963 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:58:25.177987 16027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.166682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:58865)
0620 14:58:25.167186 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.167239 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:25.168456 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:25.168936 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:25.168942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:25.169274 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:25.169721 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.169734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.170537 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.170539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.171375 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.171383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.171833 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.171835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.171984 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.172501 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:25.172516 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:25.173945 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:25.175476 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:25.175481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:25.175490 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:25.175724 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:25.176031 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:25.176033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:25.176034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:25.176153 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:25.176470 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:25.176475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:25.176672 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 14:58:25.176816 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:58:25.177046 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260620 14:58:25.179097 16025 heartbeater.cc:347] Connected to a master server at 127.7.46.190:40813
I20260620 14:58:25.179324 16025 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:25.179793 16025 heartbeater.cc:511] Master 127.7.46.190:40813 requested a full tablet report, sending...
I20260620 14:58:25.180733 15566 ts_manager.cc:194] Registered new tserver with Master: 346602ec968b4dc89b65a65cca25cfcb (127.7.46.131:45819)
I20260620 14:58:25.181233 15566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:58865
I20260620 14:58:25.190552 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab.
Entry for principal oryx/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab.
WARNING: no policy specified for oryx/127.7.46.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab.
Entry for principal oryx/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab.
WARNING: no policy specified for oryx/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab.
Entry for principal oryx/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab.
I20260620 14:58:25.259375 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 15628
I20260620 14:58:25.264935 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 15762
I20260620 14:58:25.270119 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 15896
I20260620 14:58:25.275060 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 15534
I20260620 14:58:25.280879 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:40813
--webserver_interface=127.7.46.190
--webserver_port=43509
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:25.387485 16034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:25.387722 16034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:25.387761 16034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:25.391666 16034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:25.391713 16034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:25.391732 16034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:25.391748 16034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:25.391834 16034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:25.397450 16034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:40813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=43509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:25.398396 16034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:25.399355 16034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:25.407460 16039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.407461 16042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.407477 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:25.407627 16034 server_base.cc:1034] running on GCE node
I20260620 14:58:25.408336 16034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:25.409276 16034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:25.410450 16034 hybrid_clock.cc:648] HybridClock initialized: now 1781967505410433 us; error 31 us; skew 500 ppm
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967505, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:25.413175 16034 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:58:25.413996 16034 webserver.cc:460] Webserver started at http://127.7.46.190:43509/ using document root <none> and password file <none>
I20260620 14:58:25.414503 16034 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:25.414572 16034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:25.418290 16034 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:25.420287 16049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.421439 16034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:25.421581 16034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "792ff56996db4c068552e41bdd91d4bc"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "bea3e3273416e2ce06fa1acbafb1207a"
server_key_iv: "1ad14e21694091c8a3b4af1fb596837e"
server_key_version: "encryptionkey@0"
I20260620 14:58:25.422101 16034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:25.454424 16034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:25.457490 16034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:25.457700 16034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:25.464617 16034 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:40813
I20260620 14:58:25.464643 16101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:40813 every 8 connection(s)
I20260620 14:58:25.465622 16034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:25.466362 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 16034
I20260620 14:58:25.466787 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:43673
--local_ip_for_outbound_sockets=127.7.46.129
--tserver_master_addrs=127.7.46.190:40813
--webserver_port=38483
--webserver_interface=127.7.46.129
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.46.129
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:58:25.469916 16102 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:58:25.471477 16102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Bootstrap starting.
I20260620 14:58:25.487366 16102 log.cc:824] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:25.493532 16102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:58:25.493885 16102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Bootstrap complete.
I20260620 14:58:25.497989 16102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } }
I20260620 14:58:25.498397 16102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792ff56996db4c068552e41bdd91d4bc, State: Initialized, Role: FOLLOWER
I20260620 14:58:25.498853 16102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } }
I20260620 14:58:25.498999 16102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:25.499078 16102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:25.499191 16102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:58:25.501044 16102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } }
I20260620 14:58:25.501367 16102 leader_election.cc:302] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792ff56996db4c068552e41bdd91d4bc; no voters:
I20260620 14:58:25.501654 16102 leader_election.cc:288] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:58:25.501787 16106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:58:25.502025 16106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [term 2 LEADER]: Becoming Leader. State: Replica: 792ff56996db4c068552e41bdd91d4bc, State: Running, Role: LEADER
I20260620 14:58:25.502381 16106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } }
I20260620 14:58:25.502831 16102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:25.503741 16107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "792ff56996db4c068552e41bdd91d4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } } }
I20260620 14:58:25.503907 16107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:25.504253 16108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792ff56996db4c068552e41bdd91d4bc. Latest consensus state: current_term: 2 leader_uuid: "792ff56996db4c068552e41bdd91d4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792ff56996db4c068552e41bdd91d4bc" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 40813 } } }
I20260620 14:58:25.504392 16108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:25.504706 16115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:25.507678 16115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:25.508302 16115 catalog_manager.cc:1259] Loaded cluster ID: f0fc293d740d49aabcee7d582ca38615
I20260620 14:58:25.508363 16115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:25.510607 16115 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:25.511296 16115 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 792ff56996db4c068552e41bdd91d4bc: Loaded TSK: 0
I20260620 14:58:25.512470 16115 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:58:25.576094 16104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:25.576378 16104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:25.576449 16104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:25.580425 16104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:25.580503 16104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:25.580586 16104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:25.585757 16104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.129.keytab
--principal=oryx/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:43673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=38483
--tserver_master_addrs=127.7.46.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:25.586715 16104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:25.587675 16104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:25.596623 16129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.596686 16128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.596688 16131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:25.597288 16104 server_base.cc:1034] running on GCE node
I20260620 14:58:25.597694 16104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:25.598308 16104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:25.599502 16104 hybrid_clock.cc:648] HybridClock initialized: now 1781967505599483 us; error 37 us; skew 500 ppm
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967505, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:25.602216 16104 init.cc:377] Logged in from keytab as oryx/127.7.46.129@KRBTEST.COM (short username oryx)
I20260620 14:58:25.603219 16104 webserver.cc:460] Webserver started at http://127.7.46.129:38483/ using document root <none> and password file <none>
I20260620 14:58:25.603823 16104 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:25.603956 16104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:25.607877 16104 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:25.610126 16138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.611366 16104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:58:25.611461 16104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "f0e13927c6b343feba1777075056fe6a"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "8f8e15ef0b5770eb301f8e141c46480d"
server_key_iv: "b1829932ec2f9958e7b4c09ed303a2c9"
server_key_version: "encryptionkey@0"
I20260620 14:58:25.612056 16104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:25.625736 16104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:25.628688 16104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:25.628881 16104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:25.629477 16104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:25.630301 16104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:25.630357 16104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.630394 16104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:25.630450 16104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.641114 16104 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:43673
I20260620 14:58:25.641167 16251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:43673 every 8 connection(s)
I20260620 14:58:25.642077 16104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:25.642760 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 16104
I20260620 14:58:25.643174 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:42103
--local_ip_for_outbound_sockets=127.7.46.130
--tserver_master_addrs=127.7.46.190:40813
--webserver_port=44081
--webserver_interface=127.7.46.130
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.46.130
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:58:25.655118 16255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:25.644234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:56139 (local address 127.7.46.190:40813)
0620 14:58:25.644763 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.644782 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.645220 (+ 438us) server_negotiation.cc:407] Connection header received
0620 14:58:25.646625 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.646635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.646908 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.647221 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.648385 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.649302 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.650448 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.650898 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.651071 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.654511 (+ 3440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:56139: BlockingRecv error: recv got EOF from 127.7.46.129:56139 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":109,"threads_started":1}
I20260620 14:58:25.655186 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.644382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:56139)
0620 14:58:25.645039 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.645088 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:25.646223 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:25.647291 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:25.647299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:25.647707 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:25.648202 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.648215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.649471 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.649476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.650317 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.650325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.651058 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.651062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.651250 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.651865 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:25.651881 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:25.654313 (+ 2432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":125,"threads_started":1}
W20260620 14:58:25.655582 16252 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:40813 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.46.190:40813: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:25.661190 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.656037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:54239)
0620 14:58:25.656163 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.656178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:25.656245 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:25.656473 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:25.656475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:25.656590 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:25.656670 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.656674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.657674 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.657677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.658192 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.658199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.658753 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.658755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.658831 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.659282 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:25.659293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:25.661075 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:58:25.661478 16255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:25.656120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:54239 (local address 127.7.46.190:40813)
0620 14:58:25.656226 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.656230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.656245 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:58:25.656331 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.656334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.656376 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.656449 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.656771 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.657571 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.658300 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.658642 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.658738 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.661185 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:54239: BlockingRecv error: recv got EOF from 127.7.46.129:54239 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:25.666282 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.661656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:36469)
0620 14:58:25.661774 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.661789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:25.661886 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:25.662143 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:25.662145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:25.662282 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:25.662346 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.662350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.663021 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.663027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.663521 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.663528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.664009 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.664012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.664099 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.664580 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:25.664591 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:25.666181 (+ 1590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:58:25.666419 16255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:25.661839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:36469 (local address 127.7.46.190:40813)
0620 14:58:25.661963 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.661967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.661980 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:58:25.662012 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.662014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.662044 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.662129 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.662450 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.662918 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.663633 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.663903 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.664010 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.666283 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:36469: BlockingRecv error: recv got EOF from 127.7.46.129:36469 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:58:25.666481 16252 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:58:25.752504 16254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:25.752748 16254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:25.752827 16254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:25.756912 16254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:25.756968 16254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:25.757061 16254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:25.762238 16254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.130.keytab
--principal=oryx/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:42103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=44081
--tserver_master_addrs=127.7.46.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:25.763198 16254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:25.764261 16254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:25.773554 16262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.773605 16261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.773561 16264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:25.773751 16254 server_base.cc:1034] running on GCE node
I20260620 14:58:25.774232 16254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:25.774837 16254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:25.775976 16254 hybrid_clock.cc:648] HybridClock initialized: now 1781967505775963 us; error 23 us; skew 500 ppm
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967505, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:25.778630 16254 init.cc:377] Logged in from keytab as oryx/127.7.46.130@KRBTEST.COM (short username oryx)
I20260620 14:58:25.779469 16254 webserver.cc:460] Webserver started at http://127.7.46.130:44081/ using document root <none> and password file <none>
I20260620 14:58:25.780076 16254 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:25.780169 16254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:25.783604 16254 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:25.785490 16271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.786557 16254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:25.786675 16254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "26816681acab47d1b669201c0aca7748"
format_stamp: "Formatted at 2026-06-20 14:58:24 on dist-test-slave-30t4"
server_key: "50df5e1820034db6d85f0c7e99ffbf08"
server_key_iv: "4be425d31c6691e82dbae15c64ab8067"
server_key_version: "encryptionkey@0"
I20260620 14:58:25.787159 16254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:25.806524 16254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:25.809250 16254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:25.809425 16254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:25.809965 16254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:25.810711 16254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:25.810756 16254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.810825 16254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:25.810866 16254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.819985 16254 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:42103
I20260620 14:58:25.820015 16384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:42103 every 8 connection(s)
I20260620 14:58:25.820895 16254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:25.829437 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 16254
I20260620 14:58:25.829936 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:45819
--local_ip_for_outbound_sockets=127.7.46.131
--tserver_master_addrs=127.7.46.190:40813
--webserver_port=35093
--webserver_interface=127.7.46.131
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.46.131
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:58:25.832082 16255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:25.822540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:42215 (local address 127.7.46.190:40813)
0620 14:58:25.822671 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.822677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.823453 (+ 776us) server_negotiation.cc:407] Connection header received
0620 14:58:25.824765 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.824768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.824820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.824897 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.826229 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.826781 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.827973 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.828185 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.828267 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.831960 (+ 3693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:42215: BlockingRecv error: recv got EOF from 127.7.46.130:42215 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:25.832660 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.822832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:42215)
0620 14:58:25.823291 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.823340 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:25.824566 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:25.825113 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:25.825119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:25.825527 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:25.826067 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.826088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.826898 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.826900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.827856 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.827874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.828314 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.828317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.828458 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.828993 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:25.829008 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:25.831831 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":103,"threads_started":1}
W20260620 14:58:25.833112 16385 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:40813 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.46.190:40813: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:25.839807 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.833569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:52001)
0620 14:58:25.833753 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.833768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:25.833869 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:25.834128 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:25.834134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:25.834316 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:25.834455 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.834484 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.835244 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.835250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.836307 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.836322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.836752 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.836758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.836891 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.837371 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:25.837393 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:25.839608 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260620 14:58:25.839892 16255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:25.833651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:52001 (local address 127.7.46.190:40813)
0620 14:58:25.833770 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.833776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.833825 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:58:25.833968 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.833970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.834007 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.834074 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.834640 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.835126 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.836433 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.836633 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.836713 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.839720 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:52001: BlockingRecv error: recv got EOF from 127.7.46.130:52001 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:25.845671 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.840494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:40233)
0620 14:58:25.840670 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.840687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:25.840787 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:25.841026 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:25.841031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:25.841218 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:25.841326 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.841336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.842028 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.842031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.842876 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:25.842885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.843300 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.843303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:25.843384 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.843815 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:25.843829 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:25.845561 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 14:58:25.845767 16255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:25.840589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:40233 (local address 127.7.46.190:40813)
0620 14:58:25.840704 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.840707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.840723 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:25.840880 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.840882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.840915 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.840977 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.841450 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.841921 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.842989 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.843194 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.843269 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.845665 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:40233: BlockingRecv error: recv got EOF from 127.7.46.130:40233 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 14:58:25.845965 16385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:58:25.940795 16388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:25.941052 16388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:25.941133 16388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:25.945072 16388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:25.945151 16388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:25.945236 16388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:25.950206 16388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/krb5kdc/oryx_127.7.46.131.keytab
--principal=oryx/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:45819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=35093
--tserver_master_addrs=127.7.46.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:25.951150 16388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:25.952122 16388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:25.961591 16394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:25.961632 16393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:25.961618 16388 server_base.cc:1034] running on GCE node
W20260620 14:58:25.961802 16396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:25.962208 16388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:25.962815 16388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:25.963971 16388 hybrid_clock.cc:648] HybridClock initialized: now 1781967505963930 us; error 52 us; skew 500 ppm
Jun 20 14:58:25 dist-test-slave-30t4 krb5kdc[15521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967505, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:25.966544 16388 init.cc:377] Logged in from keytab as oryx/127.7.46.131@KRBTEST.COM (short username oryx)
I20260620 14:58:25.967338 16388 webserver.cc:460] Webserver started at http://127.7.46.131:35093/ using document root <none> and password file <none>
I20260620 14:58:25.967931 16388 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:25.968029 16388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:25.971868 16388 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:25.973794 16403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:25.974949 16388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:25.975080 16388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "346602ec968b4dc89b65a65cca25cfcb"
format_stamp: "Formatted at 2026-06-20 14:58:25 on dist-test-slave-30t4"
server_key: "d1a77c518c3730735f8f1d47331d5b65"
server_key_iv: "181f6b89aafd272d20effed4acd5067b"
server_key_version: "encryptionkey@0"
I20260620 14:58:25.975589 16388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:26.001029 16388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:26.004516 16388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:26.004765 16388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:26.005439 16388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:26.006255 16388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:26.006306 16388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:26.006350 16388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:26.006374 16388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:26.015892 16388 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:45819
I20260620 14:58:26.015913 16516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:45819 every 8 connection(s)
I20260620 14:58:26.016943 16388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:26.026825 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 16388
W20260620 14:58:26.027694 16255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:26.018663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:34649 (local address 127.7.46.190:40813)
0620 14:58:26.018825 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:26.018830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:26.019691 (+ 861us) server_negotiation.cc:407] Connection header received
0620 14:58:26.020992 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:26.020995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:26.021040 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:26.021112 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:26.022210 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.022733 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.023948 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.024163 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.024249 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.027574 (+ 3325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:34649: BlockingRecv error: recv got EOF from 127.7.46.131:34649 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:26.028429 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:26.019101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:34649)
0620 14:58:26.019538 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:26.019579 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:26.020799 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:26.021286 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:26.021293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:26.021629 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:26.022084 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.022097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.022875 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.022878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.023751 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.023762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.024302 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.024304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.024441 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.024944 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:26.024959 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:26.027470 (+ 2511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
W20260620 14:58:26.028975 16517 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:40813 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.46.190:40813: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:26.034664 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:26.029392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:54077)
0620 14:58:26.029699 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:26.029714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:26.029793 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:26.030025 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:26.030027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:26.030180 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:26.030257 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.030261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.030969 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.030975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.031688 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.031696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.032162 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.032165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.032256 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.032707 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:26.032724 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:26.034585 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244}
W20260620 14:58:26.034861 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:26.029481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:54077 (local address 127.7.46.190:40813)
0620 14:58:26.029699 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:26.029704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:26.029761 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:58:26.029866 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:26.029871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:26.029919 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:26.030012 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:26.030373 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.030856 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.031830 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.032052 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.032162 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.034733 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:54077: BlockingRecv error: recv got EOF from 127.7.46.131:54077 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260620 14:58:26.037082 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:26.027950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.7.46.190:40813)
0620 14:58:26.028081 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:26.028086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:26.028101 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:58:26.028154 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:26.028156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:26.028196 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:26.028272 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:26.029113 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.029768 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.030612 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.030846 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.030955 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.032358 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:26.032365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:26.032388 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:26.032413 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:26.034337 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:26.035356 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:26.035364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:26.035373 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:26.035530 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:26.035840 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:26.035844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:26.035845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:26.036273 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:26.036464 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:26.036718 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:58:26.036969 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:26.040407 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:26.035128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:53247)
0620 14:58:26.035248 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:26.035261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:26.035333 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:26.035534 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:26.035536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:26.035707 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:26.035832 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.035842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.036717 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.036721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.037405 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.037413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.037900 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.037902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.037982 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.038397 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:26.038428 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:26.040330 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:58:26.040577 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:26.035206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:53247 (local address 127.7.46.190:40813)
0620 14:58:26.035348 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:26.035352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:26.035387 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:58:26.035417 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:26.035419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:26.035459 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:26.035530 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:26.035939 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.036607 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.037510 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.037797 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.037896 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.040465 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:53247: BlockingRecv error: recv got EOF from 127.7.46.131:53247 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 14:58:26.040616 16517 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:26.672606 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:26.667065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:52917)
0620 14:58:26.667321 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:26.667334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:26.667410 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:26.667638 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:26.667640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:26.667872 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:26.667974 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.667979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.668704 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.668707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.669360 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.669368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.669926 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.669930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.670020 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.670555 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:26.670566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:26.672441 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
W20260620 14:58:26.672694 16523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:26.667171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:52917 (local address 127.7.46.190:40813)
0620 14:58:26.667394 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:26.667397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:26.667411 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:26.667479 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:26.667481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:26.667522 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:26.667601 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:26.668079 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.668614 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.669496 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.669801 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.669905 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.672563 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:52917: BlockingRecv error: recv got EOF from 127.7.46.129:52917 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:26 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:26.851435 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:26.846532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:42851)
0620 14:58:26.846778 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:26.846792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:26.846871 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:26.847113 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:26.847116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:26.847282 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:26.847389 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.847394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.848100 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.848102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.848656 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:26.848663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.849069 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.849071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:26.849138 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.849638 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:26.849650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:26.851327 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
W20260620 14:58:26.851666 16523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:26.846668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:42851 (local address 127.7.46.190:40813)
0620 14:58:26.846812 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:26.846821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:26.846835 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:26.846964 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:26.846966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:26.847008 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:26.847077 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:26.847489 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.848014 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.848757 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:26.848972 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:26.849052 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:26.851530 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:42851: BlockingRecv error: recv got EOF from 127.7.46.130:42851 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:58:27 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:27 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:27.046329 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:27.041222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:56945)
0620 14:58:27.041471 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:27.041484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:27.041581 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:27.041845 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:27.041848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:27.042019 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:27.042110 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:27.042114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.042836 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.042839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:27.043396 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:27.043403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.043865 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.043868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:27.043927 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:27.044389 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:27.044400 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:27.046210 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
W20260620 14:58:27.046494 16523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:27.041292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:56945 (local address 127.7.46.190:40813)
0620 14:58:27.041472 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:27.041477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:27.041569 (+ 92us) server_negotiation.cc:407] Connection header received
0620 14:58:27.041679 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:27.041681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:27.041727 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:27.041794 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:27.042205 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.042715 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.043496 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.043698 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.043822 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:27.046333 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:56945: BlockingRecv error: recv got EOF from 127.7.46.131:56945 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 14:58:27 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:27 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:27.678527 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:27.673395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:51773)
0620 14:58:27.673671 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:27.673684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:27.673770 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:27.673999 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:27.674001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:27.674171 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:27.674266 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:27.674271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.674981 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.674983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:27.675524 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:27.675530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.675992 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.675995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:27.676054 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:27.676591 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:27.676602 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:27.678346 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20260620 14:58:27.678592 16527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:27.673463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:51773 (local address 127.7.46.190:40813)
0620 14:58:27.673725 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:27.673729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:27.673742 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:58:27.673851 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:27.673853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:27.673894 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:27.673964 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:27.674360 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.674891 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.675617 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.675899 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.675977 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:27.678470 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:51773: BlockingRecv error: recv got EOF from 127.7.46.129:51773 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
Jun 20 14:58:27 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:27 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:27.857928 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:27.852329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:34755)
0620 14:58:27.852634 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:27.852656 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:27.852789 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:27.853056 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:27.853059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:27.853264 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:27.853362 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:27.853367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.854102 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.854108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:27.854687 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:27.854694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.855138 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.855142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:27.855228 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:27.855735 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:27.855746 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:27.857751 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
W20260620 14:58:27.858001 16527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:27.852409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:34755 (local address 127.7.46.190:40813)
0620 14:58:27.852597 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:27.852602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:27.852731 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:58:27.852903 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:27.852906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:27.852950 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:27.853032 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:27.853479 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.853996 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.854805 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.855028 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.855134 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:27.857891 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:34755: BlockingRecv error: recv got EOF from 127.7.46.130:34755 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 14:58:28 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:28 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:28.052217 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:28.047161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:53989)
0620 14:58:28.047447 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:28.047460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:28.047549 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:28.047791 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:28.047793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:28.047968 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:28.048060 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:28.048065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.048788 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.048790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:28.049320 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:28.049326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.049713 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.049715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:28.049772 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:28.050240 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:28.050250 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:28.052084 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
W20260620 14:58:28.052290 16527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:28.047290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:53989 (local address 127.7.46.190:40813)
0620 14:58:28.047436 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:28.047440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:28.047524 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:58:28.047623 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:28.047625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:28.047668 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:28.047737 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:28.048185 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.048695 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.049417 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.049622 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.049697 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:28.052202 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:53989: BlockingRecv error: recv got EOF from 127.7.46.131:53989 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:58:28 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:28 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:28.690727 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:28.679347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:50319)
0620 14:58:28.679619 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:28.679633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:28.679720 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:28.680140 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:28.680142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:28.680311 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:28.680413 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:28.680418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.681143 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.681146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:28.681687 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:28.681693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.682083 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.682085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:28.682140 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:28.688515 (+ 6375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:28.688531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:28.690535 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
W20260620 14:58:28.690784 16531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:28.679432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:50319 (local address 127.7.46.190:40813)
0620 14:58:28.679684 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:28.679688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:28.679702 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:28.679928 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:28.679934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:28.679998 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:28.680088 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:28.680512 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.681054 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.681781 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.681993 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.682066 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:28.690658 (+ 8592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:50319: BlockingRecv error: recv got EOF from 127.7.46.129:50319 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
Jun 20 14:58:28 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:28 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:28.864178 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:28.858722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:43457)
0620 14:58:28.859016 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:28.859029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:28.859134 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:28.859384 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:28.859386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:28.859648 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:28.859747 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:28.859753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.860727 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.860730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:28.861305 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:28.861313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.861692 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.861694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:28.861754 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:28.862314 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:28.862328 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:28.864058 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":127,"threads_started":1}
W20260620 14:58:28.864391 16531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:28.858813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:43457 (local address 127.7.46.190:40813)
0620 14:58:28.858953 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:28.858957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:28.859103 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:58:28.859212 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:28.859214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:28.859257 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:28.859362 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:28.859936 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.860627 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.861403 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.861602 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.861675 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:28.864174 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:43457: BlockingRecv error: recv got EOF from 127.7.46.130:43457 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 14:58:29 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:29 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:29.060173 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.053046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:53875)
0620 14:58:29.053516 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:29.053534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:29.053649 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:29.053938 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:29.053941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:29.054178 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:29.054309 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.054317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.055304 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.055308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.056440 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.056450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.057098 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.057102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.057192 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.057730 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:29.057744 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:29.060002 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":96,"threads_started":1}
W20260620 14:58:29.060320 16531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:29.053163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:53875 (local address 127.7.46.190:40813)
0620 14:58:29.053415 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.053420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.053628 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:58:29.053742 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.053745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.053795 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.053895 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:29.054472 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.055182 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.056564 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.056976 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.057081 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.060175 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:53875: BlockingRecv error: recv got EOF from 127.7.46.131:53875 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Jun 20 14:58:29 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:29 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:29.696589 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.691405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:40951)
0620 14:58:29.691701 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:29.691722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:29.691826 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:29.692044 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:29.692046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:29.692212 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:29.692303 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.692308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.693010 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.693012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.693553 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.693560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.693961 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.693963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.694019 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.694607 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:29.694619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:29.696422 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
W20260620 14:58:29.696662 16535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:29.691501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:40951 (local address 127.7.46.190:40813)
0620 14:58:29.691808 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.691828 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.691842 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:58:29.691902 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.691904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.691944 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.692012 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:29.692400 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.692923 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.693648 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.693870 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.693943 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.696542 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:40951: BlockingRecv error: recv got EOF from 127.7.46.129:40951 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":113,"threads_started":1}
Jun 20 14:58:29 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:29 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:29.869760 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.864897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:41983)
0620 14:58:29.865159 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:29.865172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:29.865265 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:29.865477 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:29.865479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:29.865651 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:29.865744 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.865749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.866442 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.866444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.866976 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.866983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.867418 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.867421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.867484 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.867950 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:29.867960 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:29.869651 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
W20260620 14:58:29.869931 16535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:29.864944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:41983 (local address 127.7.46.190:40813)
0620 14:58:29.865075 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.865079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.865236 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:58:29.865331 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.865333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.865376 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.865447 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:29.865837 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.866353 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.867096 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.867305 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.867391 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.869772 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:41983: BlockingRecv error: recv got EOF from 127.7.46.130:41983 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 14:58:30 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:30 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:30.066529 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.061254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:35811)
0620 14:58:30.061524 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:30.061537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:30.061630 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:30.061904 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:30.061906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:30.062078 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:30.062172 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.062177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.062937 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.062939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.063480 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.063487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.063983 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.063986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.064044 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.064601 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:30.064612 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:30.066419 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
W20260620 14:58:30.066665 16535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:30.061354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:35811 (local address 127.7.46.190:40813)
0620 14:58:30.061527 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.061531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.061587 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:58:30.061717 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.061745 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.061793 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.061870 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:30.062273 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.062843 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.063611 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.063886 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.063975 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.066549 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:35811: BlockingRecv error: recv got EOF from 127.7.46.131:35811 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 14:58:30 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:30 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:30.703223 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.697514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:54943)
0620 14:58:30.697900 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:30.697916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:30.698037 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:30.698377 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:30.698381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:30.698617 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:30.698749 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.698756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.699651 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.699653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.700279 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.700295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.700733 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.700736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.700802 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.701346 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:30.701358 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:30.703115 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":105,"threads_started":1}
W20260620 14:58:30.703438 16539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:30.697645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:54943 (local address 127.7.46.190:40813)
0620 14:58:30.697878 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.697882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.698011 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:58:30.698137 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.698142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.698202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.698321 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:30.698894 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.699555 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.700402 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.700630 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.700718 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.703277 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:54943: BlockingRecv error: recv got EOF from 127.7.46.129:54943 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":53,"threads_started":1}
Jun 20 14:58:30 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:30 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:30.875998 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.870494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:32995)
0620 14:58:30.870800 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:30.870816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:30.870948 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:30.871215 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:30.871218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:30.871430 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:30.871515 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.871521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.872337 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.872342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.872913 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.872921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.873380 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.873383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.873477 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.874020 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:30.874032 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:30.875886 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
W20260620 14:58:30.876157 16539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:30.870641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:32995 (local address 127.7.46.190:40813)
0620 14:58:30.870802 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.870808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.870908 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:58:30.871051 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.871055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.871102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.871195 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:30.871630 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.872231 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.873032 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.873270 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.873378 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.876008 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:32995: BlockingRecv error: recv got EOF from 127.7.46.130:32995 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 14:58:31 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:31 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:31.072666 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:31.067436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:55195)
0620 14:58:31.067706 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:31.067721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:31.067844 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:31.068085 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:31.068087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:31.068267 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:31.068358 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:31.068363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.069048 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.069050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:31.069589 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:31.069595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.069980 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.069983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:31.070041 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.070509 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:31.070520 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:31.072501 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":115,"threads_started":1}
W20260620 14:58:31.072712 16539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:31.067528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:55195 (local address 127.7.46.190:40813)
0620 14:58:31.067663 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:31.067666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:31.067799 (+ 133us) server_negotiation.cc:407] Connection header received
0620 14:58:31.067937 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:31.067939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:31.067983 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:31.068049 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:31.068452 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.068962 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.069680 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.069890 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.069960 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.072611 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:55195: BlockingRecv error: recv got EOF from 127.7.46.131:55195 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 14:58:31 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:31 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:31.709537 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:31.704082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:50633)
0620 14:58:31.704364 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:31.704380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:31.704535 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:31.704844 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:31.704847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:31.705050 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:31.705141 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:31.705146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.705911 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.705916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:31.706470 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:31.706478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.706942 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.706946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:31.707031 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.707577 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:31.707589 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:31.709370 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
W20260620 14:58:31.709607 16543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:31.704101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:50633 (local address 127.7.46.190:40813)
0620 14:58:31.704364 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:31.704370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:31.704476 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:58:31.704633 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:31.704637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:31.704722 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:31.704820 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:31.705259 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.705801 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.706589 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.706828 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.706938 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.709485 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:50633: BlockingRecv error: recv got EOF from 127.7.46.129:50633 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
Jun 20 14:58:31 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:31 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:31.882469 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:31.876707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:50239)
0620 14:58:31.876968 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:31.876981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:31.877075 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:31.877285 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:31.877287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:31.877462 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:31.877557 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:31.877562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.878279 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.878281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:31.878896 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:31.878905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.879377 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.879380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:31.879436 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.880133 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:31.880145 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:31.882304 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260620 14:58:31.882524 16543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:31.876830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:50239 (local address 127.7.46.190:40813)
0620 14:58:31.876965 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:31.876974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:31.877028 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:58:31.877139 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:31.877141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:31.877181 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:31.877253 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:31.877652 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.878190 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.879043 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.879276 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.879369 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.882419 (+ 3050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:50239: BlockingRecv error: recv got EOF from 127.7.46.130:50239 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:58:32 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:32 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:32.078835 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.073516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:55435)
0620 14:58:32.073791 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:32.073804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:32.073921 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:32.074128 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:32.074130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:32.074307 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:32.074410 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.074415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.075209 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.075212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.075980 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.075989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.076439 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.076441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.076500 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.076941 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:32.076952 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:32.078726 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
W20260620 14:58:32.079066 16543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:32.073574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:55435 (local address 127.7.46.190:40813)
0620 14:58:32.073709 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.073714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.073896 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:58:32.073986 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.073988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.074032 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.074095 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:32.074515 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.075103 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.076119 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.076348 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.076426 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.078864 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:55435: BlockingRecv error: recv got EOF from 127.7.46.131:55435 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 14:58:32 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:32 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:32.716096 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.710332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:60475)
0620 14:58:32.710691 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:32.710713 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:32.710814 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:32.711048 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:32.711050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:32.711234 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:32.711338 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.711343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.712172 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.712174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.712732 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.712739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.713163 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.713166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.713231 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.713737 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:32.713748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:32.715906 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":86,"threads_started":1}
W20260620 14:58:32.716154 16547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:32.710453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:60475 (local address 127.7.46.190:40813)
0620 14:58:32.710690 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.710694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.710785 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:58:32.710892 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.710895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.710940 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.711016 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:32.711461 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.712081 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.712841 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.713062 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.713143 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.716036 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:60475: BlockingRecv error: recv got EOF from 127.7.46.129:60475 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":58,"threads_started":1}
Jun 20 14:58:32 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:32 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:32.888705 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.883277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:46171)
0620 14:58:32.883578 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:32.883592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:32.883692 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:32.883981 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:32.883984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:32.884159 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:32.884252 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.884258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.884969 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.884973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.885563 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.885571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.885981 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.885984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.886048 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.886604 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:32.886616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:32.888598 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
W20260620 14:58:32.888839 16547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:32.883340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:46171 (local address 127.7.46.190:40813)
0620 14:58:32.883488 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.883493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.883658 (+ 165us) server_negotiation.cc:407] Connection header received
0620 14:58:32.883810 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.883813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.883868 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.883944 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:32.884347 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.884852 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.885663 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.885869 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.885948 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.888711 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:46171: BlockingRecv error: recv got EOF from 127.7.46.130:46171 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:58:33 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:33 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:33.084985 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.079575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:37191)
0620 14:58:33.079911 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.079924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:33.080027 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:33.080317 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:33.080322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:33.080532 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:33.080622 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.080628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.081610 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.081613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.082183 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.082192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.082569 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.082571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.082631 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.083097 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:33.083108 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:33.084874 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
W20260620 14:58:33.085111 16547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:33.079652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:37191 (local address 127.7.46.190:40813)
0620 14:58:33.079807 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.079815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.079993 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:58:33.080119 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.080123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.080178 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.080266 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:33.080729 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.081508 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.082277 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.082480 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.082552 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.084994 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:37191: BlockingRecv error: recv got EOF from 127.7.46.131:37191 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 14:58:33 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:33 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:33.725890 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.716981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:51189)
0620 14:58:33.717247 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.717260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:33.717344 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:33.717572 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:33.717574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:33.717751 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:33.717843 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.717847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.718574 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.718577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.719137 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.719145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.719551 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.719554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.719615 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.724008 (+ 4393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:33.724020 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:33.725762 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
W20260620 14:58:33.726046 16551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:33.717117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:51189 (local address 127.7.46.190:40813)
0620 14:58:33.717373 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.717376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.717388 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:58:33.717422 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.717424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.717464 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.717541 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:33.717940 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.718465 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.719239 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.719460 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.719536 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.725908 (+ 6372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:51189: BlockingRecv error: recv got EOF from 127.7.46.129:51189 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
Jun 20 14:58:33 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:33 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:33.895339 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.889510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:49739)
0620 14:58:33.889791 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.889805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:33.889917 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:33.890145 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:33.890147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:33.890350 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:33.890457 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.890462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.891643 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.891650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.892463 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.892473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.892876 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.892879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.892937 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.893481 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:33.893495 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:33.895227 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
W20260620 14:58:33.895502 16551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:33.889629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:49739 (local address 127.7.46.190:40813)
0620 14:58:33.889799 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.889803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.889884 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:58:33.889998 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.890000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.890044 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.890108 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:33.890564 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.891520 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.892566 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.892772 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.892883 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.895347 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:49739: BlockingRecv error: recv got EOF from 127.7.46.130:49739 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 14:58:34 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:34 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:34.091744 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:34.085897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:54293)
0620 14:58:34.086175 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:34.086201 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:34.086329 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:34.086707 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:34.086710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:34.086947 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:34.087092 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:34.087102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.087998 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.088001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:34.088584 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:34.088591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.089043 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.089046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:34.089106 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:34.089606 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:34.089618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:34.091635 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
W20260620 14:58:34.091957 16551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:34.085870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:54293 (local address 127.7.46.190:40813)
0620 14:58:34.086046 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:34.086051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:34.086306 (+ 255us) server_negotiation.cc:407] Connection header received
0620 14:58:34.086432 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:34.086435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:34.086500 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:34.086614 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:34.087223 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.087905 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.088700 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.088946 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.089031 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:34.091837 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:54293: BlockingRecv error: recv got EOF from 127.7.46.131:54293 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:58:34 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:34 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:34.731865 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:34.726593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:40103)
0620 14:58:34.726899 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:34.726914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:34.727015 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:34.727255 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:34.727257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:34.727431 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:34.727525 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:34.727530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.728251 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.728253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:34.728796 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:34.728804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.729232 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.729234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:34.729311 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:34.729881 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:34.729893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:34.731671 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
W20260620 14:58:34.731945 16555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:34.726667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:40103 (local address 127.7.46.190:40813)
0620 14:58:34.726932 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:34.726936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:34.726982 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:58:34.727092 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:34.727094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:34.727137 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:34.727207 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:34.727614 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.728156 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.728912 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.729127 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.729204 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:34.731804 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:40103: BlockingRecv error: recv got EOF from 127.7.46.129:40103 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
Jun 20 14:58:34 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:34 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:34.903095 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:34.896196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:34233)
0620 14:58:34.896461 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:34.896484 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:34.896622 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:34.896938 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:34.896941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:34.897133 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:34.897225 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:34.897230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.897947 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.897949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:34.898866 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:34.898876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.899562 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.899566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:34.899668 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:34.900328 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:34.900342 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:34.902938 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
W20260620 14:58:34.903267 16555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:34.896236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:34233 (local address 127.7.46.190:40813)
0620 14:58:34.896370 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:34.896375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:34.896583 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:58:34.896732 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:34.896735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:34.896795 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:34.896901 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:34.897327 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.897857 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.899015 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:34.899415 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:34.899532 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:34.903125 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:34233: BlockingRecv error: recv got EOF from 127.7.46.130:34233 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 14:58:35 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:35 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:35.098120 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:35.092596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:57533)
0620 14:58:35.092862 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:35.092880 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:35.093005 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:35.093265 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:35.093268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:35.093477 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:35.093563 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:35.093568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.094296 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.094302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:35.094910 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:35.094918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.095367 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.095370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:35.095455 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:35.096036 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:35.096047 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:35.097977 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
W20260620 14:58:35.098206 16555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:35.092650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:57533 (local address 127.7.46.190:40813)
0620 14:58:35.092804 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:35.092809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:35.092955 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:58:35.093111 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:35.093116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:35.093164 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:35.093246 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:35.093677 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.094193 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.095040 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.095255 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.095358 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:35.098090 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:57533: BlockingRecv error: recv got EOF from 127.7.46.131:57533 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 14:58:35 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:35 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:35.737639 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:35.732603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:37597)
0620 14:58:35.732898 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:35.732912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:35.732994 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:35.733228 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:35.733230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:35.733404 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:35.733497 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:35.733501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.734220 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.734222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:35.734771 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:35.734778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.735171 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.735173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:35.735230 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:35.735741 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:35.735752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:35.737526 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
W20260620 14:58:35.737784 16559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:35.732806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:37597 (local address 127.7.46.190:40813)
0620 14:58:35.733031 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:35.733035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:35.733047 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:58:35.733081 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:35.733083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:35.733121 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:35.733195 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:35.733605 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.734133 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.734869 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.735080 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.735154 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:35.737648 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:37597: BlockingRecv error: recv got EOF from 127.7.46.129:37597 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
Jun 20 14:58:35 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:35 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:35.909900 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:35.904466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:34425)
0620 14:58:35.904772 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:35.904789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:35.904895 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:35.905123 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:35.905125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:35.905293 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:35.905390 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:35.905394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.906125 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.906128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:35.906709 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:35.906718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.907145 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.907149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:35.907214 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:35.907729 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:35.907741 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:35.909737 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
W20260620 14:58:35.909991 16559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:35.904531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:34425 (local address 127.7.46.190:40813)
0620 14:58:35.904676 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:35.904679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:35.904846 (+ 167us) server_negotiation.cc:407] Connection header received
0620 14:58:35.904976 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:35.904978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:35.905021 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:35.905092 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:35.905482 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.906003 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.906837 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.907041 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.907122 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:35.909873 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:34425: BlockingRecv error: recv got EOF from 127.7.46.130:34425 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:58:36 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:36 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:36.104476 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:36.098857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:36089)
0620 14:58:36.099159 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:36.099175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:36.099278 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:36.099549 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:36.099552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:36.099754 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:36.099905 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.099912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.100646 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.100653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.101215 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.101223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.101659 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.101663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.101748 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.102307 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:36.102319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:36.104307 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":97,"threads_started":1}
W20260620 14:58:36.104511 16559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:36.099017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:36089 (local address 127.7.46.190:40813)
0620 14:58:36.099159 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:36.099166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:36.099230 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:58:36.099383 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:36.099389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:36.099444 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:36.099529 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:36.100010 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.100532 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.101333 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.101554 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.101658 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.104413 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:36089: BlockingRecv error: recv got EOF from 127.7.46.131:36089 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:58:36 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:36 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:36.743840 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:36.738462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:54103)
0620 14:58:36.738738 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:36.738750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:36.738846 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:36.739091 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:36.739093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:36.739261 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:36.739362 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.739367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.740126 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.740129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.740668 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.740675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.741247 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.741251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.741348 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.741923 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:36.741935 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:36.743677 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
W20260620 14:58:36.744036 16563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:36.738516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:54103 (local address 127.7.46.190:40813)
0620 14:58:36.738759 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:36.738763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:36.738800 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:58:36.738932 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:36.738934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:36.738980 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:36.739057 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:36.739457 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.740040 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.740770 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.741107 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.741217 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.743871 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:54103: BlockingRecv error: recv got EOF from 127.7.46.129:54103 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":60,"threads_started":1}
Jun 20 14:58:36 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:36 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:36.916750 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:36.910690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:58581)
0620 14:58:36.910982 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:36.910995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:36.911125 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:36.911400 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:36.911403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:36.911610 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:36.911699 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.911705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.912877 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.912883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.913595 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.913605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.914016 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.914019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.914083 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.914576 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:36.914589 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:36.916558 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
W20260620 14:58:36.916769 16563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:36.910747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:58581 (local address 127.7.46.190:40813)
0620 14:58:36.910920 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:36.910925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:36.911069 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:58:36.911222 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:36.911226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:36.911276 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:36.911376 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:36.911859 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.912748 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.913697 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.913913 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.913992 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.916669 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:58581: BlockingRecv error: recv got EOF from 127.7.46.130:58581 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:58:37 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:37 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:37.110742 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.105152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:34833)
0620 14:58:37.105413 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.105431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.105524 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.105763 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.105765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.105942 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:37.106033 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.106038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.106796 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.106798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.107540 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.107552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.108232 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.108236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.108327 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.108854 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:37.108866 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:37.110636 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
W20260620 14:58:37.110918 16563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:37.105272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:34833 (local address 127.7.46.190:40813)
0620 14:58:37.105411 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.105414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.105497 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:58:37.105603 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.105606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.105653 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.105738 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:37.106127 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.106705 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.107709 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.108106 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.108238 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.110781 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:34833: BlockingRecv error: recv got EOF from 127.7.46.131:34833 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 14:58:37 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:37 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:37.750033 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.744643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:45623)
0620 14:58:37.744929 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.744943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.745045 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.745292 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.745294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.745465 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:37.745556 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.745560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.746320 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.746323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.746995 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.747005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.747582 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.747585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.747678 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.748301 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:37.748312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:37.749927 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
W20260620 14:58:37.750180 16567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:37.744756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:45623 (local address 127.7.46.190:40813)
0620 14:58:37.745000 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.745003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.745016 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:58:37.745140 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.745142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.745183 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.745256 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:37.745656 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.746226 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.747116 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.747466 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.747574 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.750052 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:45623: BlockingRecv error: recv got EOF from 127.7.46.129:45623 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
Jun 20 14:58:37 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:37 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:37.922575 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.917709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:41945)
0620 14:58:37.917956 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.917970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.918064 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.918262 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.918264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.918435 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:37.918528 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.918532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.919223 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.919225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.919795 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.919802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.920187 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.920189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.920245 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.920765 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:37.920776 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:37.922466 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
W20260620 14:58:37.922713 16567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:37.917763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:41945 (local address 127.7.46.190:40813)
0620 14:58:37.917921 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.917928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.918063 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:58:37.918143 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.918145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.918192 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.918253 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:37.918621 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.919130 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.919892 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.920098 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.920171 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.922599 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:41945: BlockingRecv error: recv got EOF from 127.7.46.130:41945 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:58:38 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:38 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:38.117106 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:38.111610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:54367)
0620 14:58:38.111909 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:38.111924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:38.112034 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:38.112305 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:38.112307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:38.112514 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:38.112602 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:38.112607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.113344 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.113349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:38.113903 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:38.113910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.114372 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.114376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:38.114478 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:38.115040 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:38.115051 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:38.116996 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
W20260620 14:58:38.117338 16567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:38.111808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:54367 (local address 127.7.46.190:40813)
0620 14:58:38.111959 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:38.111966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:38.111982 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:38.112141 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:38.112144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:38.112199 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:38.112286 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:38.112716 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.113235 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.114026 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.114252 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.114361 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:38.117117 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:54367: BlockingRecv error: recv got EOF from 127.7.46.131:54367 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 14:58:38 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:38 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:38.762135 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:38.750877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:35479)
0620 14:58:38.751158 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:38.751175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:38.751297 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:38.751642 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:38.751645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:38.751935 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:38.752064 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:38.752073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.753178 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.753180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:38.753974 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:38.753987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.754592 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.754596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:38.754683 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:38.760185 (+ 5502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:38.760197 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:38.761967 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260620 14:58:38.762216 16571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:38.750933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:35479 (local address 127.7.46.190:40813)
0620 14:58:38.751178 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:38.751182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:38.751233 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:58:38.751424 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:38.751427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:38.751496 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:38.751580 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:38.752184 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.753048 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.754121 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.754496 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.754588 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:38.762094 (+ 7506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:35479: BlockingRecv error: recv got EOF from 127.7.46.129:35479 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
Jun 20 14:58:38 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:38 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:38.929376 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:38.923471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:35117)
0620 14:58:38.923817 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:38.923833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:38.923985 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:38.924255 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:38.924258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:38.924455 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:38.924543 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:38.924548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.925704 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.925708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:38.926270 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:38.926278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.926746 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.926750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:38.926838 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:38.927310 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:38.927335 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:38.929196 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":122,"threads_started":1}
W20260620 14:58:38.929414 16571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:38.923647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:35117 (local address 127.7.46.190:40813)
0620 14:58:38.923815 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:38.923820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:38.923923 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:58:38.924081 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:38.924084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:38.924136 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:38.924222 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:38.924677 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.925584 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.926395 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.926629 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.926742 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:38.929317 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:35117: BlockingRecv error: recv got EOF from 127.7.46.130:35117 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:58:39 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:39 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:39.125530 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:39.117883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:38079)
0620 14:58:39.118672 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:39.118687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:39.118807 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:39.119159 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:39.119165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:39.119404 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:39.119535 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:39.119543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.120749 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.120753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:39.121925 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:39.121939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.122430 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.122432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:39.122494 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:39.123090 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:39.123105 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:39.125350 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":65,"threads_started":1}
W20260620 14:58:39.125551 16571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:39.118227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:38079 (local address 127.7.46.190:40813)
0620 14:58:39.118364 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:39.118369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:39.118748 (+ 379us) server_negotiation.cc:407] Connection header received
0620 14:58:39.118955 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:39.118958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:39.119033 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:39.119130 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:39.119669 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.120613 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.122043 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.122328 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.122411 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:39.125452 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:38079: BlockingRecv error: recv got EOF from 127.7.46.131:38079 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:58:39 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:39 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:39.768855 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:39.762885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:39975)
0620 14:58:39.763176 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:39.763189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:39.763285 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:39.763567 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:39.763570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:39.763748 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:39.763911 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:39.763917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.764735 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.764740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:39.765520 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:39.765529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.766018 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.766021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:39.766082 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:39.766644 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:39.766683 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:39.768733 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
W20260620 14:58:39.768988 16575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:39.762994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:39975 (local address 127.7.46.190:40813)
0620 14:58:39.763258 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:39.763261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:39.763274 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:58:39.763378 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:39.763380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:39.763423 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:39.763497 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:39.764012 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.764580 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.765651 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.765921 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.766000 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:39.768862 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:39975: BlockingRecv error: recv got EOF from 127.7.46.129:39975 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":59,"threads_started":1}
Jun 20 14:58:39 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:39 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:39.935101 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:39.930146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:46469)
0620 14:58:39.930431 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:39.930443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:39.930534 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:39.930758 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:39.930760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:39.930941 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:39.931033 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:39.931037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.931792 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.931795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:39.932344 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:39.932351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.932745 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.932747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:39.932807 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:39.933259 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:39.933270 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:39.934992 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
W20260620 14:58:39.935233 16575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:39.930264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:46469 (local address 127.7.46.190:40813)
0620 14:58:39.930405 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:39.930408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:39.930506 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:58:39.930611 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:39.930613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:39.930656 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:39.930724 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:39.931135 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.931668 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.932436 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.932650 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.932738 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:39.935116 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:46469: BlockingRecv error: recv got EOF from 127.7.46.130:46469 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:58:40 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:40 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.131@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:40.131662 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.126395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.131:35823)
0620 14:58:40.126683 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.126696 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.126797 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.127012 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.127014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.127192 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:40.127291 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.127296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.128010 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.128013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.128580 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.128587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.128991 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.128994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.129057 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.129605 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:40.129616 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:40.131486 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
W20260620 14:58:40.131697 16575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:40.126443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:35823 (local address 127.7.46.190:40813)
0620 14:58:40.126571 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.126575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.126768 (+ 193us) server_negotiation.cc:407] Connection header received
0620 14:58:40.126865 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.126868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.126911 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.126977 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:40.127389 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.127923 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.128676 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.128887 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.128957 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.131594 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.131:35823: BlockingRecv error: recv got EOF from 127.7.46.131:35823 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 14:58:40 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:40 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.129@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:40.774664 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.769681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.129:60833)
0620 14:58:40.769954 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.769967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.770046 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.770275 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.770277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.770443 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:40.770533 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.770538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.771235 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.771237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.771837 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.771846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.772247 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.772249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.772308 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.772756 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:40.772782 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:40.774499 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
W20260620 14:58:40.774737 16579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:40.769783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:60833 (local address 127.7.46.190:40813)
0620 14:58:40.770019 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.770022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.770035 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:58:40.770128 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.770130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.770169 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.770242 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:40.770631 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.771148 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.771943 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.772158 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.772232 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.774612 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.129:60833: BlockingRecv error: recv got EOF from 127.7.46.129:60833 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
Jun 20 14:58:40 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:58:40 dist-test-slave-30t4 krb5kdc[15521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.46.130@KRBTEST.COM for oryx/127.7.46.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:58:40.941727 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.935960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:40813 (local address 127.7.46.130:40303)
0620 14:58:40.936228 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.936241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.936333 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.936583 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.936586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.936756 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:40.936858 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.936863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.937591 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.937593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.938139 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.938146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.938637 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.938640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.938752 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.939383 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:40.939394 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:40.941491 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.46.190:40813: Server oryx/127.7.46.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
W20260620 14:58:40.941787 16579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:40.936113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:40303 (local address 127.7.46.190:40813)
0620 14:58:40.936255 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.936259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.936290 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:58:40.936428 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.936430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.936477 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.936555 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:40.936964 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.937495 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.938250 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.938501 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.938639 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.941640 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.46.130:40303: BlockingRecv error: recv got EOF from 127.7.46.130:40303 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:41.029153 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 16104
I20260620 14:58:41.034627 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 16254
I20260620 14:58:41.039662 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 16388
I20260620 14:58:41.044564 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 16034
2026-06-20T14:58:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17908 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:41Z Disabled control of system clock
I20260620 14:58:41.073863 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44161
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:44161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:58:41.179164 16584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:41.179414 16584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:41.179447 16584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:41.183387 16584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:41.183439 16584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:41.183455 16584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:41.183471 16584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:41.183486 16584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:41.188110 16584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:44161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:41.189262 16584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:41.190727 16584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:41.199539 16590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:41.199555 16589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:41.199579 16592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:41.199609 16584 server_base.cc:1034] running on GCE node
I20260620 14:58:41.200331 16584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:41.201402 16584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:41.202632 16584 hybrid_clock.cc:648] HybridClock initialized: now 1781967521202601 us; error 24 us; skew 500 ppm
I20260620 14:58:41.204690 16584 webserver.cc:460] Webserver started at http://127.7.46.190:40795/ using document root <none> and password file <none>
I20260620 14:58:41.205298 16584 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:41.205397 16584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:41.205662 16584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:41.207676 16584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "2a1f457182be4e3dad1a91c0a6019ee8"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "96edd6d925db9bbdd7b269a43bbe11db"
server_key_iv: "7d35913bd1ff7f8921189e145f42a75d"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.208329 16584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "2a1f457182be4e3dad1a91c0a6019ee8"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "96edd6d925db9bbdd7b269a43bbe11db"
server_key_iv: "7d35913bd1ff7f8921189e145f42a75d"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.211982 16584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:41.214208 16598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.215372 16584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:41.215507 16584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "2a1f457182be4e3dad1a91c0a6019ee8"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "96edd6d925db9bbdd7b269a43bbe11db"
server_key_iv: "7d35913bd1ff7f8921189e145f42a75d"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.215611 16584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:41.228586 16584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:41.229189 16584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:41.229399 16584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:41.235702 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:44161 every 8 connection(s)
I20260620 14:58:41.235707 16584 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:44161
I20260620 14:58:41.236598 16584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:41.239156 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 16584
I20260620 14:58:41.239274 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:41.239316 16651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:41.239594 7354 external_mini_cluster.cc:1428] Setting key bcc7fcf30ff1b197fd98438e11943bf1
I20260620 14:58:41.245258 16651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8: Bootstrap starting.
I20260620 14:58:41.247815 16651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:41.248574 16651 log.cc:824] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:41.248783 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.240973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.7.46.190:44161)
0620 14:58:41.241564 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:41.241578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:41.241611 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:58:41.242404 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:41.242424 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:41.242785 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:41.243110 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:41.243520 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.244500 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.245364 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.245827 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.246014 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.247106 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:41.247112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:41.247123 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:41.247475 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:41.247526 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:41.247545 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:41.247617 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:41.247837 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:58:41.248078 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260620 14:58:41.250319 16651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8: No bootstrap required, opened a new log
I20260620 14:58:41.253904 16651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1f457182be4e3dad1a91c0a6019ee8" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44161 } }
I20260620 14:58:41.254123 16651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:41.254208 16651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1f457182be4e3dad1a91c0a6019ee8, State: Initialized, Role: FOLLOWER
I20260620 14:58:41.254608 16651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1f457182be4e3dad1a91c0a6019ee8" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44161 } }
I20260620 14:58:41.254750 16651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:41.254838 16651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:41.254951 16651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:41.255692 16651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1f457182be4e3dad1a91c0a6019ee8" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44161 } }
I20260620 14:58:41.256054 16651 leader_election.cc:302] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a1f457182be4e3dad1a91c0a6019ee8; no voters:
I20260620 14:58:41.256433 16651 leader_election.cc:288] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:41.256502 16656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:41.256803 16656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1f457182be4e3dad1a91c0a6019ee8, State: Running, Role: LEADER
I20260620 14:58:41.257138 16656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1f457182be4e3dad1a91c0a6019ee8" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44161 } }
I20260620 14:58:41.257750 16651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:41.258715 16657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a1f457182be4e3dad1a91c0a6019ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1f457182be4e3dad1a91c0a6019ee8" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44161 } } }
I20260620 14:58:41.258932 16658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a1f457182be4e3dad1a91c0a6019ee8. Latest consensus state: current_term: 1 leader_uuid: "2a1f457182be4e3dad1a91c0a6019ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1f457182be4e3dad1a91c0a6019ee8" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44161 } } }
I20260620 14:58:41.259032 16657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:41.259105 16658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:41.259881 16665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:41.262291 16665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:41.267204 16665 catalog_manager.cc:1347] Generated new cluster ID: 1013c0a64f82453dbfb3493a6abf869f
I20260620 14:58:41.267269 16665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:41.277033 16665 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:41.277817 16665 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:41.281617 16665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a1f457182be4e3dad1a91c0a6019ee8: Generated new TSK 0
I20260620 14:58:41.282181 16665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:41.290930 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44161
--builtin_ntp_servers=127.7.46.148:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:58:41.403664 16675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:41.403955 16675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:41.403996 16675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:41.407945 16675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:41.407989 16675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:41.408041 16675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:41.412528 16675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:41.413405 16675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:41.414624 16675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:41.423740 16681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:41.423833 16675 server_base.cc:1034] running on GCE node
W20260620 14:58:41.423806 16683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:41.423816 16680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:41.424402 16675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:41.424975 16675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:41.426151 16675 hybrid_clock.cc:648] HybridClock initialized: now 1781967521426122 us; error 41 us; skew 500 ppm
I20260620 14:58:41.428438 16675 webserver.cc:460] Webserver started at http://127.7.46.129:43767/ using document root <none> and password file <none>
I20260620 14:58:41.429206 16675 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:41.429306 16675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:41.429569 16675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:41.432464 16675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "d687f01fbf1f46d8b86d6f4c69bbe813"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "da86419b51076c4518c0003f1f483336"
server_key_iv: "28a087f57739fc416bfef8df4e0e3808"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.433133 16675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "d687f01fbf1f46d8b86d6f4c69bbe813"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "da86419b51076c4518c0003f1f483336"
server_key_iv: "28a087f57739fc416bfef8df4e0e3808"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.437019 16675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:58:41.439321 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.440508 16675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:41.440629 16675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "d687f01fbf1f46d8b86d6f4c69bbe813"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "da86419b51076c4518c0003f1f483336"
server_key_iv: "28a087f57739fc416bfef8df4e0e3808"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.440724 16675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:41.455552 16675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:41.456192 16675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:41.456372 16675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:41.456888 16675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:41.457742 16675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:41.457787 16675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.457855 16675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:41.457886 16675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.465938 16675 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:38017
I20260620 14:58:41.466001 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:38017 every 8 connection(s)
I20260620 14:58:41.466794 16675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:41.467346 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 16675
I20260620 14:58:41.467453 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:41.467758 7354 external_mini_cluster.cc:1428] Setting key f0ac6bb17b2d466f32ea2a153562191c
I20260620 14:58:41.469887 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44161
--builtin_ntp_servers=127.7.46.148:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:58:41.476774 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.468641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:58065 (local address 127.7.46.190:44161)
0620 14:58:41.468770 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:41.468775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:41.469998 (+ 1223us) server_negotiation.cc:407] Connection header received
0620 14:58:41.472060 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:41.472064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:41.472155 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:41.472269 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:41.473109 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.473719 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.474831 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.475069 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.475178 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.476289 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:41.476293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:41.476295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:41.476308 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:41.476334 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:41.476342 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:41.476420 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:41.476576 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:58:41.476635 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:41.477537 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.469039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44161 (local address 127.7.46.129:58065)
0620 14:58:41.469823 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:41.469874 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:41.471798 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:41.472437 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:41.472444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:41.472524 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:41.472976 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:41.472988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.473840 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.473844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:41.474704 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:41.474712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.475213 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.475217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:41.475384 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.476027 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:41.476028 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:41.476065 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:41.476123 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:41.476422 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:41.476434 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:41.476440 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:58:41.476565 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:58:41.476829 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":137,"threads_started":1}
I20260620 14:58:41.478700 16803 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44161
I20260620 14:58:41.478955 16803 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:41.479571 16803 heartbeater.cc:511] Master 127.7.46.190:44161 requested a full tablet report, sending...
I20260620 14:58:41.481256 16615 ts_manager.cc:194] Registered new tserver with Master: d687f01fbf1f46d8b86d6f4c69bbe813 (127.7.46.129:38017)
I20260620 14:58:41.482731 16615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.129:58065
W20260620 14:58:41.578941 16806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:41.579176 16806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:41.579214 16806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:41.583184 16806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:41.583240 16806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:41.583292 16806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:41.587824 16806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:41.588739 16806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:41.590075 16806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:41.599112 16812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:41.599138 16814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:41.599119 16811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:41.599334 16806 server_base.cc:1034] running on GCE node
I20260620 14:58:41.599820 16806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:41.600420 16806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:41.601608 16806 hybrid_clock.cc:648] HybridClock initialized: now 1781967521601599 us; error 30 us; skew 500 ppm
I20260620 14:58:41.603288 16806 webserver.cc:460] Webserver started at http://127.7.46.130:38909/ using document root <none> and password file <none>
I20260620 14:58:41.603895 16806 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:41.603988 16806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:41.604182 16806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:41.606144 16806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "d77aec78ad6a4818a18c5f279a0c8f3c"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "a80518b52fb184149e818345d3df814f"
server_key_iv: "89f68c6fae13df740e29c91eed811211"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.606701 16806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "d77aec78ad6a4818a18c5f279a0c8f3c"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "a80518b52fb184149e818345d3df814f"
server_key_iv: "89f68c6fae13df740e29c91eed811211"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.610110 16806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:58:41.612435 16820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.613564 16806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:41.613688 16806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "d77aec78ad6a4818a18c5f279a0c8f3c"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "a80518b52fb184149e818345d3df814f"
server_key_iv: "89f68c6fae13df740e29c91eed811211"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.613792 16806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:41.626634 16806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:41.627236 16806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:41.627416 16806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:41.627991 16806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:41.628880 16806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:41.628950 16806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.629016 16806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:41.629068 16806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.637847 16806 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:40209
I20260620 14:58:41.637880 16933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:40209 every 8 connection(s)
I20260620 14:58:41.638682 16806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:41.646265 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 16806
I20260620 14:58:41.646380 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:41.646728 7354 external_mini_cluster.cc:1428] Setting key 822f329f059bae3eb4aba96ff9f5ab65
I20260620 14:58:41.647835 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.640486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:56497 (local address 127.7.46.190:44161)
0620 14:58:41.640645 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:41.640649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:41.641598 (+ 949us) server_negotiation.cc:407] Connection header received
0620 14:58:41.642940 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:41.642943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:41.642990 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:41.643069 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:41.644050 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.644623 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.645771 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.645982 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.646086 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.647302 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:41.647306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:41.647309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:41.647322 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:41.647348 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:41.647357 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:41.647434 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:41.647596 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:58:41.647658 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:58:41.648779 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.640793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44161 (local address 127.7.46.130:56497)
0620 14:58:41.641449 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:41.641488 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:41.642694 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:41.643272 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:41.643278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:41.643356 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:41.643897 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:41.643912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.644733 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.644736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:41.645644 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:41.645653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.646140 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.646147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:41.646365 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.647046 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:41.647047 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:41.647084 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:41.647142 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:41.647447 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:41.647458 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:41.647464 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:58:41.647593 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:58:41.647866 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260620 14:58:41.648814 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44161
--builtin_ntp_servers=127.7.46.148:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:58:41.649843 16934 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44161
I20260620 14:58:41.650060 16934 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:41.650645 16934 heartbeater.cc:511] Master 127.7.46.190:44161 requested a full tablet report, sending...
I20260620 14:58:41.651716 16615 ts_manager.cc:194] Registered new tserver with Master: d77aec78ad6a4818a18c5f279a0c8f3c (127.7.46.130:40209)
I20260620 14:58:41.652542 16615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.130:56497
W20260620 14:58:41.756201 16937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:41.756521 16937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:41.756575 16937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:41.760658 16937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:41.760707 16937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:41.760764 16937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:41.765342 16937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:41.766309 16937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:41.767608 16937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:41.776928 16942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:41.776927 16943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:41.777009 16945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:41.777087 16937 server_base.cc:1034] running on GCE node
I20260620 14:58:41.777647 16937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:41.778227 16937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:41.779404 16937 hybrid_clock.cc:648] HybridClock initialized: now 1781967521779380 us; error 45 us; skew 500 ppm
I20260620 14:58:41.781173 16937 webserver.cc:460] Webserver started at http://127.7.46.131:37099/ using document root <none> and password file <none>
I20260620 14:58:41.781754 16937 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:41.781845 16937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:41.782064 16937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:41.784056 16937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "acb7ae6ef4774d79bd36792351f29632"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "dc80067f8b739e8cb5916223da969304"
server_key_iv: "a4085e5412ea6046940e357409642595"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.784588 16937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "acb7ae6ef4774d79bd36792351f29632"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "dc80067f8b739e8cb5916223da969304"
server_key_iv: "a4085e5412ea6046940e357409642595"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.788043 16937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:58:41.790472 16951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.791685 16937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:41.791842 16937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "acb7ae6ef4774d79bd36792351f29632"
format_stamp: "Formatted at 2026-06-20 14:58:41 on dist-test-slave-30t4"
server_key: "dc80067f8b739e8cb5916223da969304"
server_key_iv: "a4085e5412ea6046940e357409642595"
server_key_version: "encryptionkey@0"
I20260620 14:58:41.791962 16937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:41.851013 16937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:41.851749 16937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:41.851976 16937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:41.852577 16937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:41.853523 16937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:41.853592 16937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.853662 16937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:41.853711 16937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:41.862520 16937 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:38941
I20260620 14:58:41.862548 17064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:38941 every 8 connection(s)
I20260620 14:58:41.863377 16937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:41.865947 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 16937
I20260620 14:58:41.866050 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:41.866341 7354 external_mini_cluster.cc:1428] Setting key f6aa2c55a159b4a69fbb4809f0bcb92e
I20260620 14:58:41.872840 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.865143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:43345 (local address 127.7.46.190:44161)
0620 14:58:41.865293 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:41.865297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:41.866467 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 14:58:41.868118 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:41.868121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:41.868175 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:41.868253 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:41.869154 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.869704 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.870837 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.871052 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.871157 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.872344 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:41.872348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:41.872350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:41.872362 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:41.872385 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:41.872398 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:41.872476 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:41.872630 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:58:41.872686 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:58:41.873596 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.865541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44161 (local address 127.7.46.131:43345)
0620 14:58:41.866317 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:41.866356 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:41.867928 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:41.868424 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:41.868431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:41.868539 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:41.869020 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:41.869033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.869833 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.869836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:41.870699 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:41.870707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.871189 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.871193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:41.871368 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.872083 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:41.872085 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:41.872122 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:41.872180 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:41.872478 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:41.872488 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:41.872497 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:58:41.872628 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:58:41.872884 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":178,"threads_started":1}
I20260620 14:58:41.874608 17065 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44161
I20260620 14:58:41.874809 17065 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:41.875229 17065 heartbeater.cc:511] Master 127.7.46.190:44161 requested a full tablet report, sending...
I20260620 14:58:41.876200 16615 ts_manager.cc:194] Registered new tserver with Master: acb7ae6ef4774d79bd36792351f29632 (127.7.46.131:38941)
I20260620 14:58:41.876688 16615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.131:43345
I20260620 14:58:41.880189 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:58:41.882813 16654 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:41.881909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.7.46.190:44161)
0620 14:58:41.882056 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:41.882060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:41.882217 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:58:41.882365 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:41.882368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:41.882414 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:41.882495 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:41.882715 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42480: BlockingRecv error: recv got EOF from 127.0.0.1:42480 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:58:41.883140 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 16675
I20260620 14:58:41.888561 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 16806
I20260620 14:58:41.893958 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 16937
I20260620 14:58:41.898963 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 16584
2026-06-20T14:58:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (852 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:41Z Disabled control of system clock
I20260620 14:58:41.926656 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:35039
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:35039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:58:42.031100 17076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:42.031324 17076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:42.031359 17076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:42.035300 17076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:42.035351 17076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:42.035367 17076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:42.035383 17076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:42.035399 17076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:42.039901 17076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:35039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:35039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:42.040825 17076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:42.042050 17076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:42.049947 17084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:42.049981 17081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:42.049978 17082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:42.050109 17076 server_base.cc:1034] running on GCE node
I20260620 14:58:42.050625 17076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:42.051666 17076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:42.052853 17076 hybrid_clock.cc:648] HybridClock initialized: now 1781967522052841 us; error 24 us; skew 500 ppm
I20260620 14:58:42.054525 17076 webserver.cc:460] Webserver started at http://127.7.46.190:34879/ using document root <none> and password file <none>
I20260620 14:58:42.055071 17076 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:42.055160 17076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:42.055387 17076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:42.057439 17076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "c24b811aa42f426eb35b71904e56ddab"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "5986610089414b47ca57885db4ce2057"
server_key_iv: "ff1e8b3c60a579f748852346659096b1"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.057981 17076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "c24b811aa42f426eb35b71904e56ddab"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "5986610089414b47ca57885db4ce2057"
server_key_iv: "ff1e8b3c60a579f748852346659096b1"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.061470 17076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:58:42.063719 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.064872 17076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:42.065004 17076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "c24b811aa42f426eb35b71904e56ddab"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "5986610089414b47ca57885db4ce2057"
server_key_iv: "ff1e8b3c60a579f748852346659096b1"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.065110 17076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:42.069890 17076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:42.070437 17076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:42.070607 17076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:42.077453 17076 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:35039
I20260620 14:58:42.077466 17142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:35039 every 8 connection(s)
I20260620 14:58:42.078274 17076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:42.081161 17143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:42.081477 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 17076
I20260620 14:58:42.081599 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:42.081881 7354 external_mini_cluster.cc:1428] Setting key 73ac4b2aa36b616de07da2779ee40a7d
I20260620 14:58:42.087270 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:42.083107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.7.46.190:35039)
0620 14:58:42.083706 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:42.083721 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:42.083763 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:58:42.084676 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:42.084694 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:42.085037 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:42.085368 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:42.086126 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:42.086131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:42.086141 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:42.086151 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:42.086200 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:42.086216 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:42.086356 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:42.086425 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:58:42.086646 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
I20260620 14:58:42.088043 17143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab: Bootstrap starting.
I20260620 14:58:42.090116 17143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:42.090803 17143 log.cc:824] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:42.092525 17143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab: No bootstrap required, opened a new log
I20260620 14:58:42.096055 17143 raft_consensus.cc:348] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b811aa42f426eb35b71904e56ddab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35039 } }
I20260620 14:58:42.096279 17143 raft_consensus.cc:374] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:42.096364 17143 raft_consensus.cc:729] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24b811aa42f426eb35b71904e56ddab, State: Initialized, Role: FOLLOWER
I20260620 14:58:42.096763 17143 consensus_queue.cc:260] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b811aa42f426eb35b71904e56ddab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35039 } }
I20260620 14:58:42.096907 17143 raft_consensus.cc:388] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:42.096992 17143 raft_consensus.cc:482] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:42.097105 17143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:42.097913 17143 raft_consensus.cc:504] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b811aa42f426eb35b71904e56ddab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35039 } }
I20260620 14:58:42.098228 17143 leader_election.cc:302] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c24b811aa42f426eb35b71904e56ddab; no voters:
I20260620 14:58:42.098519 17143 leader_election.cc:288] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:42.098661 17148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:42.098923 17148 raft_consensus.cc:686] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [term 1 LEADER]: Becoming Leader. State: Replica: c24b811aa42f426eb35b71904e56ddab, State: Running, Role: LEADER
I20260620 14:58:42.099195 17148 consensus_queue.cc:237] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b811aa42f426eb35b71904e56ddab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35039 } }
I20260620 14:58:42.099644 17143 sys_catalog.cc:564] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:42.100703 17150 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c24b811aa42f426eb35b71904e56ddab. Latest consensus state: current_term: 1 leader_uuid: "c24b811aa42f426eb35b71904e56ddab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b811aa42f426eb35b71904e56ddab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35039 } } }
I20260620 14:58:42.100812 17150 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:42.101115 17149 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c24b811aa42f426eb35b71904e56ddab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b811aa42f426eb35b71904e56ddab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35039 } } }
I20260620 14:58:42.101198 17157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:42.101205 17149 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:42.104183 17157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:42.109119 17157 catalog_manager.cc:1347] Generated new cluster ID: e2c9c398b0f94baf84655d5c5559a442
I20260620 14:58:42.109189 17157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:42.124136 17157 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:42.124444 17157 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:42.134500 17157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c24b811aa42f426eb35b71904e56ddab: Generated new TSK 0
I20260620 14:58:42.135118 17157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:42.142843 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35039
--builtin_ntp_servers=127.7.46.148:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:58:42.249493 17167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:42.249722 17167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:42.249760 17167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:42.253786 17167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:42.253834 17167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:42.253887 17167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:42.258570 17167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:42.259500 17167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:42.260795 17167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:42.269614 17175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:42.269639 17172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:42.269655 17173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:42.270011 17167 server_base.cc:1034] running on GCE node
I20260620 14:58:42.270372 17167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:42.270875 17167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:42.272025 17167 hybrid_clock.cc:648] HybridClock initialized: now 1781967522272003 us; error 33 us; skew 500 ppm
I20260620 14:58:42.273789 17167 webserver.cc:460] Webserver started at http://127.7.46.129:42413/ using document root <none> and password file <none>
I20260620 14:58:42.274322 17167 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:42.274420 17167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:42.274616 17167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:42.276641 17167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "de4f626e4130464c88124cde9cd8807e"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "11731656c0b3f9dee43ebef35dd22057"
server_key_iv: "db7b4376cfc0340e8c6a16a31f5bd03a"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.277186 17167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "de4f626e4130464c88124cde9cd8807e"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "11731656c0b3f9dee43ebef35dd22057"
server_key_iv: "db7b4376cfc0340e8c6a16a31f5bd03a"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.280751 17167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:42.283136 17181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.284351 17167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:42.284479 17167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "de4f626e4130464c88124cde9cd8807e"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "11731656c0b3f9dee43ebef35dd22057"
server_key_iv: "db7b4376cfc0340e8c6a16a31f5bd03a"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.284587 17167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:42.288933 17167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:42.289476 17167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:42.289662 17167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:42.290155 17167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:42.290978 17167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:42.291049 17167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.291110 17167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:42.291162 17167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.299466 17167 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:39239
I20260620 14:58:42.299490 17294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:39239 every 8 connection(s)
I20260620 14:58:42.300344 17167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:42.306192 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:42.302098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:34787 (local address 127.7.46.190:35039)
0620 14:58:42.302223 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:42.302233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:42.303159 (+ 926us) server_negotiation.cc:407] Connection header received
0620 14:58:42.304670 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:42.304676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:42.304731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:42.304828 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:42.305836 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:42.305839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:42.305841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:42.305852 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:42.305872 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:42.305880 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:42.305955 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:42.306060 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:58:42.306093 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:58:42.306977 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:42.302401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35039 (local address 127.7.46.129:34787)
0620 14:58:42.302989 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:42.303026 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:42.304410 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:42.305005 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:42.305012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:42.305092 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:42.305581 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:42.305582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:42.305618 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:42.305678 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:42.305953 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:42.305959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:42.305963 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:58:42.306080 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:58:42.306279 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20260620 14:58:42.308158 17295 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35039
I20260620 14:58:42.308399 17295 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:42.308590 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 17167
I20260620 14:58:42.308703 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:42.308837 17295 heartbeater.cc:511] Master 127.7.46.190:35039 requested a full tablet report, sending...
I20260620 14:58:42.309033 7354 external_mini_cluster.cc:1428] Setting key 3b593c7cea99d3f4ce1494d977f80a7d
I20260620 14:58:42.310269 17101 ts_manager.cc:194] Registered new tserver with Master: de4f626e4130464c88124cde9cd8807e (127.7.46.129:39239)
I20260620 14:58:42.310938 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35039
--builtin_ntp_servers=127.7.46.148:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:58:42.420413 17298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:42.420691 17298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:42.420768 17298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:42.424736 17298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:42.424815 17298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:42.424912 17298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:42.429700 17298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:42.430761 17298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:42.432191 17298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:42.441661 17303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:42.441658 17304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:42.441772 17298 server_base.cc:1034] running on GCE node
W20260620 14:58:42.441677 17306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:42.442265 17298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:42.442917 17298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:42.444088 17298 hybrid_clock.cc:648] HybridClock initialized: now 1781967522444064 us; error 30 us; skew 500 ppm
I20260620 14:58:42.445875 17298 webserver.cc:460] Webserver started at http://127.7.46.130:35241/ using document root <none> and password file <none>
I20260620 14:58:42.446447 17298 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:42.446539 17298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:42.446768 17298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:42.448812 17298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "9c3ea0940fa34f1abc7c855ed621ad25"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "41f888df668d826f99b0cad41515a3b2"
server_key_iv: "d0350831d27c40d5fd13e55e6233312a"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.449369 17298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "9c3ea0940fa34f1abc7c855ed621ad25"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "41f888df668d826f99b0cad41515a3b2"
server_key_iv: "d0350831d27c40d5fd13e55e6233312a"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.452910 17298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:42.455937 17312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.457122 17298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:42.457259 17298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "9c3ea0940fa34f1abc7c855ed621ad25"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "41f888df668d826f99b0cad41515a3b2"
server_key_iv: "d0350831d27c40d5fd13e55e6233312a"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.457378 17298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:42.461864 17298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:42.462450 17298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:42.462646 17298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:42.463163 17298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:42.464129 17298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:42.464200 17298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.464275 17298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:42.464327 17298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.473280 17298 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:44651
I20260620 14:58:42.473301 17425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:44651 every 8 connection(s)
I20260620 14:58:42.474133 17298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:42.476650 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 17298
I20260620 14:58:42.476778 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:42.477087 7354 external_mini_cluster.cc:1428] Setting key 6bd2a2f54ca7a845b39ae0fe3f3f8998
I20260620 14:58:42.479210 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35039
--builtin_ntp_servers=127.7.46.148:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:58:42.480782 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:42.475960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:36467 (local address 127.7.46.190:35039)
0620 14:58:42.476107 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:42.476112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:42.477286 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 14:58:42.478872 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:42.478875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:42.478909 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:42.478981 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:42.480274 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:42.480278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:42.480280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:42.480291 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:42.480311 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:42.480319 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:42.480390 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:42.480573 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:58:42.480624 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:58:42.481577 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:42.476358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35039 (local address 127.7.46.130:36467)
0620 14:58:42.477126 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:42.477182 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:42.478635 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:42.479177 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:42.479188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:42.479299 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:42.479975 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:42.479980 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:42.480036 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:42.480123 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:42.480401 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:42.480411 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:42.480421 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:58:42.480594 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 14:58:42.480862 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":100,"threads_started":1}
I20260620 14:58:42.482681 17426 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35039
I20260620 14:58:42.482911 17426 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:42.483287 17426 heartbeater.cc:511] Master 127.7.46.190:35039 requested a full tablet report, sending...
I20260620 14:58:42.484320 17101 ts_manager.cc:194] Registered new tserver with Master: 9c3ea0940fa34f1abc7c855ed621ad25 (127.7.46.130:44651)
W20260620 14:58:42.586802 17429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:42.587052 17429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:42.587096 17429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:42.591176 17429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:42.591238 17429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:42.591293 17429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:42.595983 17429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:42.596966 17429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:42.598248 17429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:42.607559 17434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:42.607676 17435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:42.607685 17429 server_base.cc:1034] running on GCE node
W20260620 14:58:42.607559 17437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:42.608228 17429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:42.608757 17429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:42.609913 17429 hybrid_clock.cc:648] HybridClock initialized: now 1781967522609897 us; error 29 us; skew 500 ppm
I20260620 14:58:42.611716 17429 webserver.cc:460] Webserver started at http://127.7.46.131:43819/ using document root <none> and password file <none>
I20260620 14:58:42.612437 17429 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:42.612498 17429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:42.612705 17429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:42.614689 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "77891ef00f064213b9ce52904be84878"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "f64f69d6b37b4998b396243ed136d775"
server_key_iv: "0ae80242e0a467206f8b7f3dca80372d"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.615228 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "77891ef00f064213b9ce52904be84878"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "f64f69d6b37b4998b396243ed136d775"
server_key_iv: "0ae80242e0a467206f8b7f3dca80372d"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.618882 17429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:58:42.621493 17443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.622754 17429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:58:42.622884 17429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "77891ef00f064213b9ce52904be84878"
format_stamp: "Formatted at 2026-06-20 14:58:42 on dist-test-slave-30t4"
server_key: "f64f69d6b37b4998b396243ed136d775"
server_key_iv: "0ae80242e0a467206f8b7f3dca80372d"
server_key_version: "encryptionkey@0"
I20260620 14:58:42.622991 17429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:42.628013 17429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:42.628582 17429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:42.628756 17429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:42.629287 17429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:42.630126 17429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:42.630172 17429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.630208 17429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:42.630261 17429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:42.638928 17429 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:37829
I20260620 14:58:42.639003 17556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:37829 every 8 connection(s)
I20260620 14:58:42.639860 17429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:42.645457 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 17429
I20260620 14:58:42.645562 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:42.645676 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:42.641579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:56977 (local address 127.7.46.190:35039)
0620 14:58:42.641733 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:42.641737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:42.642681 (+ 944us) server_negotiation.cc:407] Connection header received
0620 14:58:42.644031 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:42.644034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:42.644078 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:42.644153 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:42.645169 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:42.645172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:42.645175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:42.645188 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:42.645212 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:42.645222 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:42.645315 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:42.645483 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:58:42.645532 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:58:42.645898 7354 external_mini_cluster.cc:1428] Setting key dc6543fc995163b299bc0e14fb1cfd5f
I20260620 14:58:42.646744 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:42.641946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35039 (local address 127.7.46.131:56977)
0620 14:58:42.642540 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:42.642577 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:42.643818 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:42.644304 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:42.644310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:42.644391 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:42.644914 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:42.644916 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:42.644953 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:42.645016 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:42.645328 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:42.645336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:42.645342 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:58:42.645499 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 14:58:42.645796 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":95,"threads_started":1}
I20260620 14:58:42.648001 17557 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35039
I20260620 14:58:42.648234 17557 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:42.648636 17557 heartbeater.cc:511] Master 127.7.46.190:35039 requested a full tablet report, sending...
I20260620 14:58:42.649602 17101 ts_manager.cc:194] Registered new tserver with Master: 77891ef00f064213b9ce52904be84878 (127.7.46.131:37829)
I20260620 14:58:42.659747 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:58:42.662611 17146 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:58:42.661785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.7.46.190:35039)
0620 14:58:42.661934 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:42.661939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:42.662152 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:58:42.662221 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:42.662223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:42.662254 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:42.662339 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:42.662520 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51642: BlockingRecv error: recv got EOF from 127.0.0.1:51642 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:58:42.662904 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 17167
I20260620 14:58:42.668222 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 17298
I20260620 14:58:42.673120 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 17429
I20260620 14:58:42.679415 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 17076
2026-06-20T14:58:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (780 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:42 dist-test-slave-30t4 krb5kdc[17567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:42 dist-test-slave-30t4 krb5kdc[17567](info): set up 2 sockets
Jun 20 14:58:42 dist-test-slave-30t4 krb5kdc[17567](info): commencing operation
krb5kdc: starting...
W20260620 14:58:43.716851 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:43 dist-test-slave-30t4 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:58:43.832794 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:35171
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:35171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:43.939340 17580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:43.939604 17580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:43.939677 17580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:43.943588 17580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:43.943671 17580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:43.943709 17580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:43.943746 17580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:43.943796 17580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:43.948825 17580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:35171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:35171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:43.949753 17580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:43.950708 17580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:43.959013 17585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:43.959023 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:43.959235 17580 server_base.cc:1034] running on GCE node
W20260620 14:58:43.959028 17586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:43.959896 17580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:43.960916 17580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:43.962077 17580 hybrid_clock.cc:648] HybridClock initialized: now 1781967523962066 us; error 25 us; skew 500 ppm
Jun 20 14:58:43 dist-test-slave-30t4 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:43.968152 17580 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:58:43.969029 17580 webserver.cc:460] Webserver started at http://127.7.46.190:38041/ using document root <none> and password file <none>
I20260620 14:58:43.969552 17580 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:43.969645 17580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:43.969870 17580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:43.971851 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "64a68397ea5d4b8da35616a207c454ec"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-30t4"
server_key: "d0b08261597212bf60a53fd827992940"
server_key_iv: "841cdfc3c667ca12747a490c1e5820fb"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.972407 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "64a68397ea5d4b8da35616a207c454ec"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-30t4"
server_key: "d0b08261597212bf60a53fd827992940"
server_key_iv: "841cdfc3c667ca12747a490c1e5820fb"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.975950 17580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:43.978264 17595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:43.979409 17580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:43.979547 17580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "64a68397ea5d4b8da35616a207c454ec"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-30t4"
server_key: "d0b08261597212bf60a53fd827992940"
server_key_iv: "841cdfc3c667ca12747a490c1e5820fb"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.979652 17580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:43.989421 17580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:43.991986 17580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:43.992177 17580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:43.998826 17580 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:35171
I20260620 14:58:43.999111 17647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:35171 every 8 connection(s)
I20260620 14:58:43.999750 17580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:44.002358 17648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:44.007470 17648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec: Bootstrap starting.
I20260620 14:58:44.008026 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 17580
I20260620 14:58:44.008119 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:58:44.008415 7354 external_mini_cluster.cc:1428] Setting key fa9aa84b735838954a8f15f20db3036a
I20260620 14:58:44.010183 17648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:44.010871 17648 log.cc:824] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:44.012933 17648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec: No bootstrap required, opened a new log
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:44.017278 17648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a68397ea5d4b8da35616a207c454ec" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35171 } }
I20260620 14:58:44.017530 17648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:44.017612 17648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a68397ea5d4b8da35616a207c454ec, State: Initialized, Role: FOLLOWER
I20260620 14:58:44.018052 17648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a68397ea5d4b8da35616a207c454ec" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35171 } }
I20260620 14:58:44.018219 17648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:44.018292 17648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:44.018399 17648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:44.019551 17648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a68397ea5d4b8da35616a207c454ec" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35171 } }
I20260620 14:58:44.019935 17648 leader_election.cc:302] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a68397ea5d4b8da35616a207c454ec; no voters:
I20260620 14:58:44.020258 17648 leader_election.cc:288] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:44.020385 17653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:44.020598 17653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [term 1 LEADER]: Becoming Leader. State: Replica: 64a68397ea5d4b8da35616a207c454ec, State: Running, Role: LEADER
I20260620 14:58:44.020970 17653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a68397ea5d4b8da35616a207c454ec" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35171 } }
I20260620 14:58:44.021543 17648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:44.022151 17655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a68397ea5d4b8da35616a207c454ec. Latest consensus state: current_term: 1 leader_uuid: "64a68397ea5d4b8da35616a207c454ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a68397ea5d4b8da35616a207c454ec" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35171 } } }
I20260620 14:58:44.022380 17655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:44.022588 17654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a68397ea5d4b8da35616a207c454ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a68397ea5d4b8da35616a207c454ec" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 35171 } } }
I20260620 14:58:44.022756 17654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:44.022828 17662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:44.023864 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.009687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.7.46.190:35171)
0620 14:58:44.010104 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.010119 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.010151 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:58:44.010799 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.010816 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.011167 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.011536 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:44.012372 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.013400 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.014576 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.015045 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.015156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.018046 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:44.018051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:44.018068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:44.018101 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:44.020206 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.020775 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.020780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.020786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.020920 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.021237 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.021241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.021243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.021733 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:44.021960 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.022627 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.022952 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260620 14:58:44.025885 17662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:44.031211 17662 catalog_manager.cc:1347] Generated new cluster ID: 12554388450f4216ad2a2a67f814de9e
I20260620 14:58:44.031307 17662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:44.045941 17662 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:44.047125 17662 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:44.053220 17662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec: Generated new TSK 0
I20260620 14:58:44.053942 17662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:58:44.088135 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35171
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:44.193782 17674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:44.194023 17674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:44.194064 17674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:44.198055 17674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:44.198107 17674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:44.198186 17674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:58:44.203341 17674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:44.204386 17674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:44.205374 17674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:44.214596 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.214596 17679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.214602 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:44.214941 17674 server_base.cc:1034] running on GCE node
I20260620 14:58:44.215310 17674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:44.215986 17674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:44.217149 17674 hybrid_clock.cc:648] HybridClock initialized: now 1781967524217131 us; error 27 us; skew 500 ppm
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:44.219935 17674 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:58:44.220759 17674 webserver.cc:460] Webserver started at http://127.7.46.129:34831/ using document root <none> and password file <none>
I20260620 14:58:44.221309 17674 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:44.221419 17674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:44.221659 17674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:44.223932 17674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "93afc9c8b89e462ea0dd8556f21c48c0"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "19d1976c56a47b18fa59d136b078400e"
server_key_iv: "432ffe38d8a781f6a134b50deb5f0532"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.224485 17674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "93afc9c8b89e462ea0dd8556f21c48c0"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "19d1976c56a47b18fa59d136b078400e"
server_key_iv: "432ffe38d8a781f6a134b50deb5f0532"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.228003 17674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:44.230187 17689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.231336 17674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:44.231453 17674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "93afc9c8b89e462ea0dd8556f21c48c0"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "19d1976c56a47b18fa59d136b078400e"
server_key_iv: "432ffe38d8a781f6a134b50deb5f0532"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.231547 17674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:44.254384 17674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:44.257087 17674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:44.257275 17674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:44.257817 17674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:44.258571 17674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:44.258616 17674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.258679 17674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:44.258709 17674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.267575 17674 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:39787
I20260620 14:58:44.267602 17802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:39787 every 8 connection(s)
I20260620 14:58:44.268445 17674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:44.273640 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 17674
I20260620 14:58:44.273754 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:44.274022 7354 external_mini_cluster.cc:1428] Setting key 33fbbd467c8e5132d073fb1c9a526a24
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:44.282413 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.270105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:36721 (local address 127.7.46.190:35171)
0620 14:58:44.270225 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.270229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.270965 (+ 736us) server_negotiation.cc:407] Connection header received
0620 14:58:44.272344 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.272347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.272417 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.272509 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:44.273622 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.274325 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.275817 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.276044 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.276123 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.278890 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:44.278897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:44.278903 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:44.278933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:44.280482 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.280990 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.280995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.280997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.281074 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.281419 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.281422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.281424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.281591 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:44.281696 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.282140 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.282253 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 14:58:44.283286 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.270408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35171 (local address 127.7.46.129:36721)
0620 14:58:44.270827 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.270866 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.272132 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.272671 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.272678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.273014 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:44.273477 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.273490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.274487 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.274490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.275655 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.275669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.276196 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.276202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.276436 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.277076 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:44.277101 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:44.278731 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:44.280613 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:44.280618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:44.280632 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:44.280883 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:44.281171 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:44.281174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:44.281176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:44.281314 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:44.281694 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:44.281699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:44.281917 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.282102 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.282474 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260620 14:58:44.284513 17803 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35171
I20260620 14:58:44.284777 17803 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 14:58:44.285322 17803 heartbeater.cc:511] Master 127.7.46.190:35171 requested a full tablet report, sending...
I20260620 14:58:44.286841 17612 ts_manager.cc:194] Registered new tserver with Master: 93afc9c8b89e462ea0dd8556f21c48c0 (127.7.46.129:39787)
I20260620 14:58:44.288354 17612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:36721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:58:44.299686 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35171
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:44.405436 17808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:44.405646 17808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:44.405685 17808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:44.409550 17808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:44.409595 17808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:44.409684 17808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:58:44.414666 17808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:44.415557 17808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:44.416489 17808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:44.425956 17816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.425985 17814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.425995 17813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:44.426040 17808 server_base.cc:1034] running on GCE node
I20260620 14:58:44.426555 17808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:44.427116 17808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:44.428267 17808 hybrid_clock.cc:648] HybridClock initialized: now 1781967524428252 us; error 25 us; skew 500 ppm
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:44.430720 17808 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:58:44.431459 17808 webserver.cc:460] Webserver started at http://127.7.46.130:45675/ using document root <none> and password file <none>
I20260620 14:58:44.432020 17808 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:44.432080 17808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:44.432300 17808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:44.434175 17808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "6c9b4f5d841947be8ff46fefc98c5e22"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "a8dcdbf2e5231432001e9fdf933e5506"
server_key_iv: "da9dc22cb0f703d355544cc976a98204"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.434782 17808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "6c9b4f5d841947be8ff46fefc98c5e22"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "a8dcdbf2e5231432001e9fdf933e5506"
server_key_iv: "da9dc22cb0f703d355544cc976a98204"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.438246 17808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:58:44.440399 17823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.441496 17808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:58:44.441622 17808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "6c9b4f5d841947be8ff46fefc98c5e22"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "a8dcdbf2e5231432001e9fdf933e5506"
server_key_iv: "da9dc22cb0f703d355544cc976a98204"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.441735 17808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:44.461046 17808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:44.463649 17808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:44.463863 17808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:44.464388 17808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:44.465145 17808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:44.465211 17808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.465276 17808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:44.465322 17808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.474282 17808 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:37269
I20260620 14:58:44.474306 17936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:37269 every 8 connection(s)
I20260620 14:58:44.475109 17808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:44.485262 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 17808
I20260620 14:58:44.485353 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:44.485726 7354 external_mini_cluster.cc:1428] Setting key 82f6f1d8cf093e182a34b5f5b9147f2c
I20260620 14:58:44.488332 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.476796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:37431 (local address 127.7.46.190:35171)
0620 14:58:44.476963 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.476970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.477739 (+ 769us) server_negotiation.cc:407] Connection header received
0620 14:58:44.479100 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.479103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.479160 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.479264 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:44.480400 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.480926 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.482048 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.482261 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.482357 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.484700 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:44.484703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:44.484705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:44.484727 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:44.486270 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.486764 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.486767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.486769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.486847 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.487234 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.487236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.487237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.487399 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:44.487472 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.488126 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.488222 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:58:44.489328 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.477179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35171 (local address 127.7.46.130:37431)
0620 14:58:44.477601 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.477641 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.478897 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.479425 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.479432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.479758 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:44.480264 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.480277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.481062 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.481065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.481902 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.481910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.482396 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.482399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.482529 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.483129 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:44.483145 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:44.484561 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:44.486386 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:44.486391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:44.486404 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:44.486667 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:44.486941 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:44.486948 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:44.486951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:44.487138 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:44.487502 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:44.487509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:44.487849 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.488069 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.488394 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260620 14:58:44.490345 17937 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35171
I20260620 14:58:44.490576 17937 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:44.491003 17937 heartbeater.cc:511] Master 127.7.46.190:35171 requested a full tablet report, sending...
I20260620 14:58:44.492007 17612 ts_manager.cc:194] Registered new tserver with Master: 6c9b4f5d841947be8ff46fefc98c5e22 (127.7.46.130:37269)
I20260620 14:58:44.492861 17612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:37431
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:58:44.511157 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35171
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:44.617218 17942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:44.617434 17942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:44.617473 17942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:44.621323 17942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:44.621369 17942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:44.621424 17942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:58:44.626685 17942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:35171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:58:44.627521 17942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:44.628450 17942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:58:44.637472 17942 server_base.cc:1034] running on GCE node
W20260620 14:58:44.637660 17950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.637725 17947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.637682 17948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:44.638209 17942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:44.638769 17942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:44.639946 17942 hybrid_clock.cc:648] HybridClock initialized: now 1781967524639936 us; error 28 us; skew 500 ppm
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:44.642474 17942 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:58:44.643236 17942 webserver.cc:460] Webserver started at http://127.7.46.131:42773/ using document root <none> and password file <none>
I20260620 14:58:44.643751 17942 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:44.643894 17942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:44.644109 17942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:44.646063 17942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "3c9754e50750422b9d9ba081657b5282"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "0c477d7708055f0571360503b7bc5f12"
server_key_iv: "e6cb317d4f4940dec41aeed23146ab5e"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.646590 17942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "3c9754e50750422b9d9ba081657b5282"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "0c477d7708055f0571360503b7bc5f12"
server_key_iv: "e6cb317d4f4940dec41aeed23146ab5e"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.650067 17942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:58:44.652314 17957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.653445 17942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:58:44.653570 17942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "3c9754e50750422b9d9ba081657b5282"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-30t4"
server_key: "0c477d7708055f0571360503b7bc5f12"
server_key_iv: "e6cb317d4f4940dec41aeed23146ab5e"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.653672 17942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:44.674530 17942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:44.677312 17942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:44.677568 17942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:44.678071 17942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:44.678763 17942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:44.678807 17942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.678843 17942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:44.678858 17942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.687269 17942 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:36777
I20260620 14:58:44.687299 18070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:36777 every 8 connection(s)
I20260620 14:58:44.688077 17942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:44.696568 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 17942
I20260620 14:58:44.696691 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:44.696961 7354 external_mini_cluster.cc:1428] Setting key 266d575d222f752f5b1c2f299d967538
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:58:44.700908 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.689742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:41521 (local address 127.7.46.190:35171)
0620 14:58:44.689875 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.689879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.690624 (+ 745us) server_negotiation.cc:407] Connection header received
0620 14:58:44.691916 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.691918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.691963 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.692034 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:44.693054 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.693562 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.694636 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.694851 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.694950 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.697653 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:44.697656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:44.697658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:44.697685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:44.699232 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.699647 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.699650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.699651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.699744 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.700067 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.700069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.700070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.700229 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:44.700298 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.700709 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.700797 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:58:44.701651 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.690065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:35171 (local address 127.7.46.131:41521)
0620 14:58:44.690478 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.690518 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.691685 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.692190 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.692196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.692511 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:44.692933 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.692946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.693664 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.693667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.694526 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.694535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.694977 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.694979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.695106 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.695707 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:44.695729 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:44.697507 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:44.699329 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:44.699333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:44.699343 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:44.699569 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:44.699865 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:44.699868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:44.699870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:44.699995 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:44.700306 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:44.700311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:44.700506 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.700643 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.700872 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260620 14:58:44.702661 18071 heartbeater.cc:347] Connected to a master server at 127.7.46.190:35171
I20260620 14:58:44.702881 18071 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:44.703320 18071 heartbeater.cc:511] Master 127.7.46.190:35171 requested a full tablet report, sending...
I20260620 14:58:44.704289 17612 ts_manager.cc:194] Registered new tserver with Master: 3c9754e50750422b9d9ba081657b5282 (127.7.46.131:36777)
I20260620 14:58:44.704785 17612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:41521
I20260620 14:58:44.710695 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:44.718818 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.712445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.7.46.190:35171)
0620 14:58:44.712573 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.712577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.712751 (+ 174us) server_negotiation.cc:407] Connection header received
0620 14:58:44.712844 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.712846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.712881 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.712942 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:44.713513 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.713971 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.714735 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.714926 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.714998 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.716286 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:44.716288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:44.716290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:44.716318 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:44.717467 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.717820 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.717822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.717823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.717886 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.718101 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.718102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.718104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.718273 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:44.718373 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.718549 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.718631 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:58:44.721262 17612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:44.724239 17612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:44.738036 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.732227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:37269 (local address 127.0.0.1:40320)
0620 14:58:44.733251 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.733264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.733363 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.733805 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.733808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.733826 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:44.733923 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.733932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.735126 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.735133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.736754 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.736760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.737529 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.737532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.737631 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.737664 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.737765 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.737879 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":106,"thread_start_us":80,"threads_started":1}
I20260620 14:58:44.738185 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.732195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.7.46.131:36777)
0620 14:58:44.733018 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.733032 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.733055 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:58:44.733124 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.733129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.733324 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.733469 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:44.733867 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.735033 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.736912 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.737684 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.737781 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.737906 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.737964 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.738035 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":122,"threads_started":1}
I20260620 14:58:44.738579 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.732871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.7.46.129:39787)
0620 14:58:44.733548 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.733561 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.734151 (+ 590us) server_negotiation.cc:407] Connection header received
0620 14:58:44.734223 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.734227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.734402 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.734534 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:44.734758 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.735784 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.736868 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.737673 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.737781 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.737906 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.738337 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.738436 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":119,"threads_started":1}
I20260620 14:58:44.738612 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.733098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.7.46.130:37269)
0620 14:58:44.733463 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.733473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.733491 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:58:44.733555 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.733558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.733675 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.733822 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:44.734038 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.735013 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.736875 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.737409 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.737530 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.737670 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.738392 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.738482 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":90,"threads_started":1}
I20260620 14:58:44.738194 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.732066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:36777 (local address 127.0.0.1:57834)
0620 14:58:44.732555 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.732606 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.732742 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.733637 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.733641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.733660 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:44.733757 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.733761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.735200 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.735203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.736737 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.736748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.737778 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.737781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.737859 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.737877 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.737935 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.737987 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":115,"thread_start_us":63,"threads_started":1}
I20260620 14:58:44.738473 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.732712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:39787 (local address 127.0.0.1:49776)
0620 14:58:44.733978 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.733990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.734066 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.734534 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.734537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.734555 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:44.734631 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.734637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.735926 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.735929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.736744 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.736754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.738091 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.738097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.738195 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.738215 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.738282 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.738358 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":25,"thread_start_us":92,"threads_started":1}
I20260620 14:58:44.741801 17737 tablet_service.cc:1467] Processing CreateTablet for tablet e6b44ef788b042d3b9ac8c19e097baf1 (DEFAULT_TABLE table=test-table [id=8072e82640ee4beba0e36e152950d3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:44.741772 18005 tablet_service.cc:1467] Processing CreateTablet for tablet e6b44ef788b042d3b9ac8c19e097baf1 (DEFAULT_TABLE table=test-table [id=8072e82640ee4beba0e36e152950d3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:44.741772 17871 tablet_service.cc:1467] Processing CreateTablet for tablet e6b44ef788b042d3b9ac8c19e097baf1 (DEFAULT_TABLE table=test-table [id=8072e82640ee4beba0e36e152950d3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:44.742777 17737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b44ef788b042d3b9ac8c19e097baf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:44.742789 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b44ef788b042d3b9ac8c19e097baf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:44.742782 17871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b44ef788b042d3b9ac8c19e097baf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:44.750778 18089 tablet_bootstrap.cc:492] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0: Bootstrap starting.
I20260620 14:58:44.751006 18088 tablet_bootstrap.cc:492] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: Bootstrap starting.
I20260620 14:58:44.752135 18090 tablet_bootstrap.cc:492] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: Bootstrap starting.
I20260620 14:58:44.753142 18089 tablet_bootstrap.cc:654] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:44.753594 18088 tablet_bootstrap.cc:654] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:44.754560 18090 tablet_bootstrap.cc:654] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:44.754532 18088 log.cc:824] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:44.755259 18089 log.cc:824] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:44.755388 18090 log.cc:824] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:44.756619 18088 tablet_bootstrap.cc:492] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: No bootstrap required, opened a new log
I20260620 14:58:44.756817 18088 ts_tablet_manager.cc:1397] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 14:58:44.757284 18089 tablet_bootstrap.cc:492] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0: No bootstrap required, opened a new log
I20260620 14:58:44.757458 18089 ts_tablet_manager.cc:1397] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:58:44.757491 18090 tablet_bootstrap.cc:492] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: No bootstrap required, opened a new log
I20260620 14:58:44.757642 18090 ts_tablet_manager.cc:1397] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:58:44.760264 18089 raft_consensus.cc:348] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.760536 18089 raft_consensus.cc:374] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:44.760586 18089 raft_consensus.cc:729] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93afc9c8b89e462ea0dd8556f21c48c0, State: Initialized, Role: FOLLOWER
I20260620 14:58:44.760540 18088 raft_consensus.cc:348] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.760778 18088 raft_consensus.cc:374] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:44.760825 18088 raft_consensus.cc:729] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9b4f5d841947be8ff46fefc98c5e22, State: Initialized, Role: FOLLOWER
I20260620 14:58:44.760993 18089 consensus_queue.cc:260] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.761219 18088 consensus_queue.cc:260] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.761224 18090 raft_consensus.cc:348] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.761598 17803 heartbeater.cc:503] Master 127.7.46.190:35171 was elected leader, sending a full tablet report...
I20260620 14:58:44.761940 18089 ts_tablet_manager.cc:1428] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 14:58:44.762113 18088 ts_tablet_manager.cc:1428] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:58:44.762389 17937 heartbeater.cc:503] Master 127.7.46.190:35171 was elected leader, sending a full tablet report...
I20260620 14:58:44.761430 18090 raft_consensus.cc:374] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:44.762774 18090 raft_consensus.cc:729] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9754e50750422b9d9ba081657b5282, State: Initialized, Role: FOLLOWER
I20260620 14:58:44.763060 18090 consensus_queue.cc:260] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.763654 18071 heartbeater.cc:503] Master 127.7.46.190:35171 was elected leader, sending a full tablet report...
I20260620 14:58:44.764115 18090 ts_tablet_manager.cc:1428] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 14:58:44.769522 17804 tablet.cc:2369] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:44.812374 18096 raft_consensus.cc:482] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:44.812561 18096 raft_consensus.cc:504] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.813709 18096 leader_election.cc:288] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c9b4f5d841947be8ff46fefc98c5e22 (127.7.46.130:37269), 93afc9c8b89e462ea0dd8556f21c48c0 (127.7.46.129:39787)
I20260620 14:58:44.817915 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.813930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:37269 (local address 127.7.46.131:36563)
0620 14:58:44.814382 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.814397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.814485 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.814811 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.814826 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.814848 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:44.814986 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.814995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.815876 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.815880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.816783 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.816792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.817499 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.817502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.817623 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.817659 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.817730 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.817802 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
I20260620 14:58:44.818009 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.814084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:36563 (local address 127.7.46.130:37269)
0620 14:58:44.814239 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.814243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.814575 (+ 332us) server_negotiation.cc:407] Connection header received
0620 14:58:44.814633 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.814636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.814686 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.814772 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:44.815121 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.815742 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.816909 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.817386 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.817476 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.817515 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.817783 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.817859 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:58:44.818176 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.814233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:39787 (local address 127.7.46.131:58821)
0620 14:58:44.814687 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.814700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.814787 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.815122 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.815125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.815143 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:44.815248 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.815254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.816183 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.816186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.817019 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.817029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.817758 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.817762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.817859 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.817882 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.817972 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.818041 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":129,"threads_started":1}
I20260620 14:58:44.818219 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.814461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:58821 (local address 127.7.46.129:39787)
0620 14:58:44.814610 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.814614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.814775 (+ 161us) server_negotiation.cc:407] Connection header received
0620 14:58:44.814919 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.814922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.814972 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.815063 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:44.815366 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.816078 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.817135 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.817575 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.817689 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.817757 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.818002 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.818082 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:58:44.818773 17891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b44ef788b042d3b9ac8c19e097baf1" candidate_uuid: "3c9754e50750422b9d9ba081657b5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" is_pre_election: true
I20260620 14:58:44.818943 17757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b44ef788b042d3b9ac8c19e097baf1" candidate_uuid: "3c9754e50750422b9d9ba081657b5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" is_pre_election: true
I20260620 14:58:44.819069 17891 raft_consensus.cc:2454] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9754e50750422b9d9ba081657b5282 in term 0.
I20260620 14:58:44.819257 17757 raft_consensus.cc:2454] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9754e50750422b9d9ba081657b5282 in term 0.
I20260620 14:58:44.819595 17959 leader_election.cc:302] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9754e50750422b9d9ba081657b5282, 6c9b4f5d841947be8ff46fefc98c5e22; no voters:
I20260620 14:58:44.819869 18096 raft_consensus.cc:2781] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:44.819974 18096 raft_consensus.cc:482] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:44.820034 18096 raft_consensus.cc:3037] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:44.821020 18096 raft_consensus.cc:504] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.821420 18096 leader_election.cc:288] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [CANDIDATE]: Term 1 election: Requested vote from peers 6c9b4f5d841947be8ff46fefc98c5e22 (127.7.46.130:37269), 93afc9c8b89e462ea0dd8556f21c48c0 (127.7.46.129:39787)
I20260620 14:58:44.821806 17891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b44ef788b042d3b9ac8c19e097baf1" candidate_uuid: "3c9754e50750422b9d9ba081657b5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9b4f5d841947be8ff46fefc98c5e22"
I20260620 14:58:44.821859 17757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b44ef788b042d3b9ac8c19e097baf1" candidate_uuid: "3c9754e50750422b9d9ba081657b5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93afc9c8b89e462ea0dd8556f21c48c0"
I20260620 14:58:44.821954 17891 raft_consensus.cc:3037] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:44.821975 17757 raft_consensus.cc:3037] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:44.823091 17757 raft_consensus.cc:2454] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9754e50750422b9d9ba081657b5282 in term 1.
I20260620 14:58:44.823148 17891 raft_consensus.cc:2454] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9754e50750422b9d9ba081657b5282 in term 1.
I20260620 14:58:44.823372 17959 leader_election.cc:302] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9754e50750422b9d9ba081657b5282, 93afc9c8b89e462ea0dd8556f21c48c0; no voters:
I20260620 14:58:44.823523 18096 raft_consensus.cc:2781] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:44.823828 18096 raft_consensus.cc:686] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9754e50750422b9d9ba081657b5282, State: Running, Role: LEADER
I20260620 14:58:44.824157 18096 consensus_queue.cc:237] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } }
I20260620 14:58:44.827046 17610 catalog_manager.cc:5495] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c9754e50750422b9d9ba081657b5282 (127.7.46.131). New cstate: current_term: 1 leader_uuid: "3c9754e50750422b9d9ba081657b5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9754e50750422b9d9ba081657b5282" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 } health_report { overall_health: UNKNOWN } } }
I20260620 14:58:44.841151 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.837693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.7.46.131:36777)
0620 14:58:44.837811 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.837818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.837850 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:58:44.838037 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.838040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.838088 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.838202 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:44.838438 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.838925 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.839790 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.840001 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.840104 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.840401 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:44.840758 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:44.840888 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.840950 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.840999 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:58:44.846552 17891 raft_consensus.cc:1261] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 1 FOLLOWER]: Refusing update from remote peer 3c9754e50750422b9d9ba081657b5282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:44.846552 17757 raft_consensus.cc:1261] T e6b44ef788b042d3b9ac8c19e097baf1 P 93afc9c8b89e462ea0dd8556f21c48c0 [term 1 FOLLOWER]: Refusing update from remote peer 3c9754e50750422b9d9ba081657b5282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:44.847199 18096 consensus_queue.cc:1035] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9b4f5d841947be8ff46fefc98c5e22" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:44.847368 18098 consensus_queue.cc:1035] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93afc9c8b89e462ea0dd8556f21c48c0" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:44.854106 18101 mvcc.cc:205] Tried to move back new op lower bound from 7298938981764935680 to 7298938981682196480. Current Snapshot: MvccSnapshot[applied={T|T < 7298938981764935680}]
I20260620 14:58:44.854827 18102 mvcc.cc:205] Tried to move back new op lower bound from 7298938981764935680 to 7298938981682196480. Current Snapshot: MvccSnapshot[applied={T|T < 7298938981764935680}]
I20260620 14:58:44.862182 17610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:44.862314 17610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:44.864552 17610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a68397ea5d4b8da35616a207c454ec: Sending DeleteTablet for 3 replicas of tablet e6b44ef788b042d3b9ac8c19e097baf1
I20260620 14:58:44.865240 18005 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b44ef788b042d3b9ac8c19e097baf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:44 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:57834
I20260620 14:58:44.865290 17871 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b44ef788b042d3b9ac8c19e097baf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:44 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:40320
I20260620 14:58:44.865285 17737 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b44ef788b042d3b9ac8c19e097baf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:44 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:49776
I20260620 14:58:44.865675 18112 tablet_replica.cc:330] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: stopping tablet replica
I20260620 14:58:44.865978 18112 raft_consensus.cc:2229] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:44.866287 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 17674
I20260620 14:58:44.866338 18112 raft_consensus.cc:2258] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:44.867295 18111 tablet_replica.cc:330] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: stopping tablet replica
I20260620 14:58:44.867568 18111 raft_consensus.cc:2229] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:44.867861 18111 raft_consensus.cc:2258] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:44.869016 18111 ts_tablet_manager.cc:1905] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:44.869015 18112 ts_tablet_manager.cc:1905] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:44.871562 18111 ts_tablet_manager.cc:1918] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:44.871562 18112 ts_tablet_manager.cc:1918] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:44.871654 18112 log.cc:1196] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/e6b44ef788b042d3b9ac8c19e097baf1
I20260620 14:58:44.871726 18111 log.cc:1196] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/e6b44ef788b042d3b9ac8c19e097baf1
I20260620 14:58:44.872083 18112 ts_tablet_manager.cc:1939] T e6b44ef788b042d3b9ac8c19e097baf1 P 3c9754e50750422b9d9ba081657b5282: Deleting consensus metadata
I20260620 14:58:44.872133 18111 ts_tablet_manager.cc:1939] T e6b44ef788b042d3b9ac8c19e097baf1 P 6c9b4f5d841947be8ff46fefc98c5e22: Deleting consensus metadata
I20260620 14:58:44.873206 17596 catalog_manager.cc:4841] TS 3c9754e50750422b9d9ba081657b5282 (127.7.46.131:36777): tablet e6b44ef788b042d3b9ac8c19e097baf1 (table test-table [id=8072e82640ee4beba0e36e152950d3e7]) successfully deleted
I20260620 14:58:44.873214 17597 catalog_manager.cc:4841] TS 6c9b4f5d841947be8ff46fefc98c5e22 (127.7.46.130:37269): tablet e6b44ef788b042d3b9ac8c19e097baf1 (table test-table [id=8072e82640ee4beba0e36e152950d3e7]) successfully deleted
W20260620 14:58:44.875415 17597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:39787 (error 108)
I20260620 14:58:44.876127 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 17808
I20260620 14:58:44.876531 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.876175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:39787 (local address 127.0.0.1:49780)
0620 14:58:44.876372 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.876440 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 14:58:44.877621 17597 catalog_manager.cc:4584] TS 93afc9c8b89e462ea0dd8556f21c48c0 (127.7.46.129:39787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6b44ef788b042d3b9ac8c19e097baf1: Network error: Client connection negotiation failed: client connection to 127.7.46.129:39787: connect: Connection refused (error 111)
I20260620 14:58:44.881939 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 17942
I20260620 14:58:44.887353 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 17580
2026-06-20T14:58:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2208 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[18117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[18117](info): set up 2 sockets
Jun 20 14:58:44 dist-test-slave-30t4 krb5kdc[18117](info): commencing operation
krb5kdc: starting...
W20260620 14:58:45.920737 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:58:45 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.46.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.46.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/rangeradmin_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/rangeradmin_127.7.46.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.46.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.46.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/rangerlookup_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/rangerlookup_127.7.46.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.46.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.46.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/HTTP_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/HTTP_127.7.46.148@KRBTEST.COM.keytab.
I20260620 14:58:46.079474 7354 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLLAc44/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/postgres -l logfile start
2026-06-20 14:58:48.400 UTC [18147] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:58:48.400 UTC [18147] LOG: listening on IPv4 address "127.7.46.148", port 43257
2026-06-20 14:58:48.403 UTC [18147] LOG: listening on Unix socket "/tmp/.s.PGSQL.43257"
2026-06-20 14:58:48.424 UTC [18150] LOG: database system was shut down at 2026-06-20 14:58:47 UTC
2026-06-20 14:58:48.427 UTC [18147] LOG: database system is ready to accept connections
I20260620 14:58:49.396826 7354 mini_postgres.cc:96] Postgres bound to 43257
2026-06-20 14:58:49.402 UTC [18158] FATAL: database "slave" does not exist
127.7.46.148:43257 - accepting connections
I20260620 14:58:49.403595 7354 mini_ranger.cc:161] Starting Ranger...
I20260620 14:58:49.417824 7354 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:58:49.842770 7354 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:58:49.842862 7354 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-admin
2026-06-20 14:58:50,101 [I] DB FLAVOR :POSTGRES
2026-06-20 14:58:50,102 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:58:50,102 [I] Checking connection..
2026-06-20 14:58:50,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:58:50,339 [I] Checking connection passed.
2026-06-20 14:58:50,340 [I] --------- Verifying version history table ---------
2026-06-20 14:58:50,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:58:50,579 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:58:50,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:58:50,806 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:58:50,807 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:58:50,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:58:51.048 UTC [18222] WARNING: there is no transaction in progress
2026-06-20 14:58:51,054 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:58:51,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:58:51,278 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:58:51,279 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:58:51,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:58:51,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:58:51,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-30t4.c.gcp-upstream.internal','N') ;"
2026-06-20 14:58:52,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:58:52,236 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:58:52,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:58:52,479 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:58:52,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:58:52,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:58:52,939 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:58:52,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:58:53.813 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.827 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.842 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.856 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.873 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.891 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.946 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.952 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.965 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.974 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.983 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:53.996 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:54.005 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:54.010 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:54.016 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:54.022 UTC [18330] WARNING: there is no transaction in progress
2026-06-20 14:58:54,578 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:58:54,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:58:54,818 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:58:54,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:58:55,042 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:58:55,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:58:55,296 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:58:55,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:58:55,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:58:55,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-30t4.c.gcp-upstream.internal';"
2026-06-20 14:58:55,975 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:58:55,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:58:56,224 [I] DB_PATCHES have already been applied
I20260620 14:58:56.229624 7354 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLLAc44/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:58:56.229717 7354 mini_ranger.cc:193] Using host: 127.7.46.148
I20260620 14:58:56.233055 7354 mini_ranger.cc:227] Ranger admin URL: http://127.7.46.148:59231
Jun 20, 2026 2:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:58:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:58:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:58:57.350979 7354 mini_ranger.cc:160] Time spent starting Ranger: real 7.947s user 0.001s sys 0.008s
Jun 20, 2026 2:59:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:59:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 2:59:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:59:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:59:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:59:18.375249 7354 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.46.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.46.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/rangerkms_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/rangerkms_127.7.46.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.46.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.46.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.46.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/HTTP_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.46.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/HTTP_127.7.46.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:59:18 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967558, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:59:18.459363 7354 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:59:18.468391 7354 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:59:19.282554 7354 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:59:19.282650 7354 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms
I20260620 14:59:19.282749 7354 mini_ranger_kms.cc:227] Using postgres at 127.7.46.148:43257
2026-06-20 14:59:19,486 [I] DB FLAVOR :POSTGRES
2026-06-20 14:59:19,486 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:59:19,486 [I] Checking connection
2026-06-20 14:59:19,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:59:19,739 [I] connection success
2026-06-20 14:59:19,739 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:59:19,739 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:59:19,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:59:19,995 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:59:19,995 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:59:19,995 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:59:19,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:43257/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:59:20,291 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:59:20.750691 7354 mini_ranger_kms.cc:310] Created kms service
I20260620 14:59:20.948418 7354 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:59:20.990810 7354 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:59:20 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967558, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 14:59:21.350772 7354 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:59:21.350924 7354 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:59:21.350948 7354 mini_ranger_kms.cc:240] Using host: 127.7.46.148
I20260620 14:59:21.355000 7354 mini_ranger_kms.cc:276] Ranger KMS PID: 18510
I20260620 14:59:21.355122 7354 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.46.148:39297
Jun 20, 2026 2:59:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:59:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:59:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:59:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:59:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:59:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:59:23.579968 7354 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.121s user 0.000s sys 0.185s
I20260620 14:59:23.580096 7354 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:59:23.580200 7354 mini_ranger_kms.cc:401] 127.7.46.148:39297/kms/v1/keys
Jun 20 14:59:23 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.46.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:59:24 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.46.148@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
Jun 20 14:59:24 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.46.148@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
Jun 20 14:59:26 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 14:59:26.656500 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34583
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:34583
--ranger_config_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:39297/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:26.768682 18561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:26.768918 18561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:26.768956 18561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:26.772979 18561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:26.773032 18561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:26.773051 18561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:26.773092 18561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:26.773120 18561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:26.778318 18561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.46.148:39297/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:34583
--ranger_config_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:59:26.779261 18561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:26.780292 18561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:26.789206 18569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:26.789206 18567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:26.789258 18561 server_base.cc:1034] running on GCE node
W20260620 14:59:26.789198 18566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:26.789991 18561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:26.791078 18561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:26.792263 18561 hybrid_clock.cc:648] HybridClock initialized: now 1781967566792234 us; error 51 us; skew 500 ppm
Jun 20 14:59:26 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967566, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:26.797952 18561 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 14:59:26.798781 18561 webserver.cc:460] Webserver started at http://127.7.46.190:41557/ using document root <none> and password file <none>
I20260620 14:59:26.799302 18561 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:26.799381 18561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:26.799600 18561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:59:26 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967566, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 14:59:26.850311 18561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "a61a47262c734e3583b70018ab82aa45"
format_stamp: "Formatted at 2026-06-20 14:59:26 on dist-test-slave-30t4"
server_key: "aebda272072df0b0e2ad6b4488c99989"
server_key_iv: "855c0449fb828f50858e7e13b0ba47ae"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:26.850931 18561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "a61a47262c734e3583b70018ab82aa45"
format_stamp: "Formatted at 2026-06-20 14:59:26 on dist-test-slave-30t4"
server_key: "aebda272072df0b0e2ad6b4488c99989"
server_key_iv: "855c0449fb828f50858e7e13b0ba47ae"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:26.855110 18561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:59:26.869032 18576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:26.870409 18561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:59:26.870558 18561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "a61a47262c734e3583b70018ab82aa45"
format_stamp: "Formatted at 2026-06-20 14:59:26 on dist-test-slave-30t4"
server_key: "aebda272072df0b0e2ad6b4488c99989"
server_key_iv: "855c0449fb828f50858e7e13b0ba47ae"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:26.870682 18561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:26.881872 18561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:26.884608 18561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:26.884790 18561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:26.892318 18561 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:34583
I20260620 14:59:26.892336 18628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:34583 every 8 connection(s)
I20260620 14:59:26.893250 18561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:26.895898 18629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:26.900983 18629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45: Bootstrap starting.
I20260620 14:59:26.903152 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 18561
I20260620 14:59:26.903250 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 14:59:26.903371 18629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:26.903554 7354 external_mini_cluster.cc:1428] Setting key 849788582d07da9ac887416ea2e3b3a3
I20260620 14:59:26.904203 18629 log.cc:824] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:26.906747 18629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45: No bootstrap required, opened a new log
I20260620 14:59:26.910820 18629 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a47262c734e3583b70018ab82aa45" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34583 } }
I20260620 14:59:26.911078 18629 raft_consensus.cc:374] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:26.911175 18629 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61a47262c734e3583b70018ab82aa45, State: Initialized, Role: FOLLOWER
I20260620 14:59:26.911651 18629 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a47262c734e3583b70018ab82aa45" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34583 } }
I20260620 14:59:26.911810 18629 raft_consensus.cc:388] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:26.911876 18629 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:26.911989 18629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:26.912801 18629 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a47262c734e3583b70018ab82aa45" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34583 } }
I20260620 14:59:26.913120 18629 leader_election.cc:302] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a61a47262c734e3583b70018ab82aa45; no voters:
Jun 20 14:59:26 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:59:26.913486 18629 leader_election.cc:288] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:26.913559 18634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:26.913807 18634 raft_consensus.cc:686] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [term 1 LEADER]: Becoming Leader. State: Replica: a61a47262c734e3583b70018ab82aa45, State: Running, Role: LEADER
I20260620 14:59:26.914155 18634 consensus_queue.cc:237] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a47262c734e3583b70018ab82aa45" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34583 } }
I20260620 14:59:26.914846 18629 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:26.915865 18636 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a61a47262c734e3583b70018ab82aa45. Latest consensus state: current_term: 1 leader_uuid: "a61a47262c734e3583b70018ab82aa45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a47262c734e3583b70018ab82aa45" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34583 } } }
I20260620 14:59:26.915977 18636 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:26.916296 18635 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a61a47262c734e3583b70018ab82aa45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a47262c734e3583b70018ab82aa45" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34583 } } }
I20260620 14:59:26.916388 18635 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:26.916705 18643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:26.917162 18629 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:59:26.922308 18643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:26.922883 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:26.904676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.7.46.190:34583)
0620 14:59:26.905103 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:26.905121 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:26.905160 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:59:26.906371 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:26.906393 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:26.906766 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:26.907122 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:26.907849 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:26.909420 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:26.910241 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:26.910887 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:26.911057 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:26.913899 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:26.913906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:26.913933 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:26.913962 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:26.915941 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:26.916456 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:26.916465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:26.916473 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:26.916614 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:26.916953 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:26.916958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:26.916963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:26.917438 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:26.917629 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:26.918333 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 14:59:26.920757 (+ 2424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":112,"threads_started":1}
I20260620 14:59:26.929713 18643 catalog_manager.cc:1347] Generated new cluster ID: a8e4dc86772847e994a71959fc556bae
I20260620 14:59:26.929816 18643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:26.955646 18643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:26.957000 18643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:26.962531 18643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45: Generated new TSK 0
I20260620 14:59:26.963392 18643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 14:59:27.600344 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:39297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34583
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:27.734134 18674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:27.734436 18674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:27.734556 18674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:27.738708 18674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:27.738780 18674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:27.738885 18674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 14:59:27.745743 18674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.46.148:39297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:59:27.747078 18674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:27.748430 18674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:27.758983 18681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:27.759400 18674 server_base.cc:1034] running on GCE node
W20260620 14:59:27.759555 18682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:27.759409 18684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:27.760196 18674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:27.760982 18674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:27.762171 18674 hybrid_clock.cc:648] HybridClock initialized: now 1781967567762142 us; error 44 us; skew 500 ppm
Jun 20 14:59:27 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:27.765760 18674 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 14:59:27.766934 18674 webserver.cc:460] Webserver started at http://127.7.46.129:34257/ using document root <none> and password file <none>
I20260620 14:59:27.767668 18674 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:27.767812 18674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:27.768064 18674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:59:27 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 14:59:27.788965 18674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239"
format_stamp: "Formatted at 2026-06-20 14:59:27 on dist-test-slave-30t4"
server_key: "669d47f66eabae3c6449d0da46ef3e67"
server_key_iv: "dd4312cfe4e91d73d900d53cb413d34d"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:27.789710 18674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239"
format_stamp: "Formatted at 2026-06-20 14:59:27 on dist-test-slave-30t4"
server_key: "669d47f66eabae3c6449d0da46ef3e67"
server_key_iv: "dd4312cfe4e91d73d900d53cb413d34d"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:27.794196 18674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:59:27.812121 18691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:27.813807 18674 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 14:59:27.813936 18674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239"
format_stamp: "Formatted at 2026-06-20 14:59:27 on dist-test-slave-30t4"
server_key: "669d47f66eabae3c6449d0da46ef3e67"
server_key_iv: "dd4312cfe4e91d73d900d53cb413d34d"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:27.814075 18674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:27.831255 18674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:27.840929 18674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:27.841167 18674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:27.841881 18674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:27.842928 18674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:27.842999 18674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:27.843097 18674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:27.843153 18674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:27.855679 18674 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:38051
I20260620 14:59:27.855981 18804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:38051 every 8 connection(s)
I20260620 14:59:27.856911 18674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 14:59:27.857789 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 18674
I20260620 14:59:27.857915 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 14:59:27.858251 7354 external_mini_cluster.cc:1428] Setting key 4cb76ddc448184164e63faf06cc5144d
Jun 20 14:59:27 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 14:59:27.876199 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.859560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:57303 (local address 127.7.46.190:34583)
0620 14:59:27.859860 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:27.859869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:27.861701 (+ 1832us) server_negotiation.cc:407] Connection header received
0620 14:59:27.862251 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:27.862254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:27.862302 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:27.862375 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:27.864185 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:27.864771 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:27.866629 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:27.867053 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:27.867853 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:27.870932 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:27.870937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:27.870940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:27.870968 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:27.872948 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:27.873562 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:27.873565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:27.873567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:27.873657 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:27.874065 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:27.874069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:27.874071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:27.874296 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:27.874392 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:27.875896 (+ 1504us) server_negotiation.cc:299] Negotiation successful
0620 14:59:27.876020 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":120,"threads_started":1}
I20260620 14:59:27.877169 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.860078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34583 (local address 127.7.46.129:57303)
0620 14:59:27.860552 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:27.860603 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:27.862016 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:27.862911 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:27.862931 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:27.863294 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:27.864014 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:27.864037 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:27.864907 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:27.864914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:27.866473 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:27.866487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:27.867658 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:27.867664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:27.867911 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:27.868610 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:27.868633 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:27.870728 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:27.873083 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:27.873089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:27.873105 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:27.873442 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:27.873760 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:27.873763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:27.873765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:27.873935 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:27.874460 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:27.874466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:27.874768 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 14:59:27.874968 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 14:59:27.875279 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":128,"threads_started":1}
I20260620 14:59:27.878734 18805 heartbeater.cc:347] Connected to a master server at 127.7.46.190:34583
I20260620 14:59:27.878969 18805 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:27.879530 18805 heartbeater.cc:511] Master 127.7.46.190:34583 requested a full tablet report, sending...
I20260620 14:59:27.881206 18593 ts_manager.cc:194] Registered new tserver with Master: ffcf72ed5b8b4fbd9aacb2a7297d8239 (127.7.46.129:38051)
I20260620 14:59:27.882735 18593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:57303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 14:59:27.906697 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:39297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34583
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:59:27 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:59:28.030210 18813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:28.030489 18813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:28.030565 18813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:28.035374 18813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:28.035460 18813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:28.035562 18813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 14:59:28.041177 18813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.46.148:39297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:59:28.042219 18813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:28.043286 18813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:28.052860 18819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:28.053038 18820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:28.052855 18822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:28.054121 18813 server_base.cc:1034] running on GCE node
I20260620 14:59:28.054536 18813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:28.055172 18813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:28.056352 18813 hybrid_clock.cc:648] HybridClock initialized: now 1781967568056318 us; error 45 us; skew 500 ppm
Jun 20 14:59:28 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:28.059018 18813 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 14:59:28.059993 18813 webserver.cc:460] Webserver started at http://127.7.46.130:37003/ using document root <none> and password file <none>
I20260620 14:59:28.060611 18813 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:28.060779 18813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:28.061064 18813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:59:28 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 14:59:28.076310 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "3431696f380e45e880d7ac7f768e9fc8"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-30t4"
server_key: "c421ddb19337aab647eeb40875ca2a5f"
server_key_iv: "1d405176c539263e1722fef72352a5ca"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:28.076999 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "3431696f380e45e880d7ac7f768e9fc8"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-30t4"
server_key: "c421ddb19337aab647eeb40875ca2a5f"
server_key_iv: "1d405176c539263e1722fef72352a5ca"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:28.081719 18813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 14:59:28.096271 18829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:28.097679 18813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:59:28.097818 18813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "3431696f380e45e880d7ac7f768e9fc8"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-30t4"
server_key: "c421ddb19337aab647eeb40875ca2a5f"
server_key_iv: "1d405176c539263e1722fef72352a5ca"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:28.097951 18813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:28.117094 18813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:28.120859 18813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:28.121115 18813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:28.121867 18813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:28.122993 18813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:28.123090 18813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:28.123176 18813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:28.123225 18813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:28.134624 18813 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:35253
I20260620 14:59:28.134865 18942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:35253 every 8 connection(s)
I20260620 14:59:28.135816 18813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 14:59:28.145844 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 18813
I20260620 14:59:28.145998 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 14:59:28.146283 7354 external_mini_cluster.cc:1428] Setting key ee0bf79bb91d809c6dc49e225fe00075
Jun 20 14:59:28 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:59:28.155649 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.138281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34583 (local address 127.7.46.130:56775)
0620 14:59:28.139416 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.139470 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.140546 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.141277 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.141288 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.142079 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.142866 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.142889 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.144096 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.144099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.145508 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.145518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.146282 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.146286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.146484 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.148235 (+ 1751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.148254 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.150008 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.152316 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.152322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.152337 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.152695 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.153034 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.153037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.153039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.153215 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.153731 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.153740 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.154076 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.154291 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.154614 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":131,"threads_started":1}
I20260620 14:59:28.156250 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.137808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:56775 (local address 127.7.46.190:34583)
0620 14:59:28.137954 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.137959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.139571 (+ 1612us) server_negotiation.cc:407] Connection header received
0620 14:59:28.140792 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.140797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.140850 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.140963 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.143071 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.143954 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.145639 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.146012 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.146118 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.150201 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.150205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.150208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.150234 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.152188 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.152814 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.152817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.152819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.152910 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.153347 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.153351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.153353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.153582 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.153677 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.155931 (+ 2254us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.156073 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:59:28.157655 18943 heartbeater.cc:347] Connected to a master server at 127.7.46.190:34583
I20260620 14:59:28.157910 18943 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:28.158447 18943 heartbeater.cc:511] Master 127.7.46.190:34583 requested a full tablet report, sending...
I20260620 14:59:28.159581 18593 ts_manager.cc:194] Registered new tserver with Master: 3431696f380e45e880d7ac7f768e9fc8 (127.7.46.130:35253)
I20260620 14:59:28.160367 18593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:56775
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 14:59:28.183269 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:39297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34583
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:28.344023 18948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:28.344412 18948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:28.344502 18948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:28.350426 18948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:28.350505 18948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:28.350589 18948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 14:59:28.358110 18948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.46.148:39297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:59:28.359573 18948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:28.360976 18948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:28.373484 18956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:28.373689 18953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:28.373512 18954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:28.374137 18948 server_base.cc:1034] running on GCE node
I20260620 14:59:28.374735 18948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:28.375512 18948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:28.376734 18948 hybrid_clock.cc:648] HybridClock initialized: now 1781967568376690 us; error 55 us; skew 500 ppm
Jun 20 14:59:28 dist-test-slave-30t4 krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:28.380271 18948 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 14:59:28.381364 18948 webserver.cc:460] Webserver started at http://127.7.46.131:42137/ using document root <none> and password file <none>
I20260620 14:59:28.382071 18948 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:28.382167 18948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:28.382452 18948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:59:28 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 14:59:28.397468 18948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "2570dd4de07948449ae4515edb8e89a8"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-30t4"
server_key: "9676832b461be080d703e40689723002"
server_key_iv: "c91a8d01a93ee68be85b4c826e4ed3b4"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:28.398195 18948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "2570dd4de07948449ae4515edb8e89a8"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-30t4"
server_key: "9676832b461be080d703e40689723002"
server_key_iv: "c91a8d01a93ee68be85b4c826e4ed3b4"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:28.425529 18948 fs_manager.cc:691] Time spent creating directory manager: real 0.027s user 0.006s sys 0.000s
Jun 20 14:59:28 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 14:59:28.442448 18963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:28.444483 18948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:59:28.444666 18948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "2570dd4de07948449ae4515edb8e89a8"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-30t4"
server_key: "9676832b461be080d703e40689723002"
server_key_iv: "c91a8d01a93ee68be85b4c826e4ed3b4"
server_key_version: "kuduclusterkey@0"
I20260620 14:59:28.444875 18948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:28.469848 18948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:28.473538 18948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:28.473815 18948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:28.474669 18948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:28.475840 18948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:28.475934 18948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:28.476008 18948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:28.476084 18948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:28.487869 18948 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:44683
I20260620 14:59:28.487883 19076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:44683 every 8 connection(s)
I20260620 14:59:28.489017 18948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 14:59:28.494190 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 18948
I20260620 14:59:28.494318 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 14:59:28.494616 7354 external_mini_cluster.cc:1428] Setting key bc5ca9016c31caaafd29ce2ca3581a28
Jun 20 14:59:28 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 14:59:28.505656 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.493918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:59903 (local address 127.7.46.190:34583)
0620 14:59:28.494066 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.494070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.494092 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:59:28.495227 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.495231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.495287 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.495340 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.497021 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.497769 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.499426 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.499634 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.499752 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.502151 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.502155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.502158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.502184 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.503676 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.504192 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.504197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.504199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.504281 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.504630 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.504633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.504634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.504810 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.504910 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.505383 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.505493 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:59:28.506531 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.492831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34583 (local address 127.7.46.131:59903)
0620 14:59:28.493861 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.493925 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.494976 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.495743 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.495754 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.496156 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.496856 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.496877 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.497912 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.497915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.499306 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.499317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.499823 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.499828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.499992 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.500593 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.500610 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.501973 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.503815 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.503822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.503837 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.504089 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.504379 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.504382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.504384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.504527 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.504919 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.504928 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.505154 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.505331 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.505627 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":145,"threads_started":1}
I20260620 14:59:28.507674 19077 heartbeater.cc:347] Connected to a master server at 127.7.46.190:34583
I20260620 14:59:28.507961 19077 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:28.508432 19077 heartbeater.cc:511] Master 127.7.46.190:34583 requested a full tablet report, sending...
I20260620 14:59:28.509400 18591 ts_manager.cc:194] Registered new tserver with Master: 2570dd4de07948449ae4515edb8e89a8 (127.7.46.131:44683)
I20260620 14:59:28.509958 18591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:59903
I20260620 14:59:28.521324 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:59:28.531489 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.523381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.7.46.190:34583)
0620 14:59:28.523507 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.523512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.523739 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:59:28.523889 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.523892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.523936 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.524030 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.524711 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.525297 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.526106 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.526331 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.526446 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.528431 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.528434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.528437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.528461 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.529894 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.530316 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.530321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.530323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.530399 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.530668 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.530673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.530675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.530842 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.530983 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.531213 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.531324 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:59:28.534421 18591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:59:28.538210 18591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:59:28.554152 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.547218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:35253 (local address 127.0.0.1:58468)
0620 14:59:28.547921 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.547982 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.548112 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.549305 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.549308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.549341 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:28.549539 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.549547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.551029 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.551033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.552496 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.552507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.553621 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.553624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.553732 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.553777 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.553864 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.553979 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":43,"spinlock_wait_cycles":3712,"thread_start_us":86,"threads_started":1}
I20260620 14:59:28.554584 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.547962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:44683 (local address 127.0.0.1:52440)
0620 14:59:28.549525 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.549537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.549617 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.550393 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.550396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.550411 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:28.550499 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.550504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.551734 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.551737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.553149 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.553159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.554213 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.554217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.554307 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.554327 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.554377 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.554442 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1508,"mutex_wait_us":144,"thread_start_us":83,"threads_started":1}
I20260620 14:59:28.555204 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.548355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.7.46.131:44683)
0620 14:59:28.549855 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.549866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.549886 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:59:28.549967 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.549971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.550137 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.550266 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:28.550616 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.551615 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.553306 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.554076 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.554746 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.554889 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.554944 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.555059 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1403,"thread_start_us":160,"threads_started":1}
I20260620 14:59:28.555714 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.548386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.7.46.130:35253)
0620 14:59:28.548829 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.548841 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.548864 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:59:28.548941 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.548946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.549139 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.549686 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:28.549820 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.550901 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.552658 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.553488 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.555013 (+ 1525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.555141 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.555483 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.555576 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":143,"threads_started":1}
I20260620 14:59:28.556205 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.547268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:38051 (local address 127.0.0.1:48320)
0620 14:59:28.548339 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.548351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.548427 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.549280 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.549286 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.549308 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:28.549517 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.549524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.551004 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.551007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.553149 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.553159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.555163 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.555166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.555269 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.555302 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.556000 (+ 698us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.556077 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":131,"spinlock_wait_cycles":896,"thread_start_us":101,"threads_started":1}
I20260620 14:59:28.556710 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.548065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.7.46.129:38051)
0620 14:59:28.548783 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.548791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.548814 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:59:28.548888 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.548893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.549119 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.549294 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:28.549650 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.550885 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.553288 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.554076 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.554190 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.555449 (+ 1259us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.556486 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.556582 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":68,"threads_started":1}
I20260620 14:59:28.558286 19011 tablet_service.cc:1467] Processing CreateTablet for tablet 850e0887d8094dfc8c6d336ee961c1cb (DEFAULT_TABLE table=test-table [id=ced59ba780eb4b10aa5871da402c4a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:59:28.558267 18877 tablet_service.cc:1467] Processing CreateTablet for tablet 850e0887d8094dfc8c6d336ee961c1cb (DEFAULT_TABLE table=test-table [id=ced59ba780eb4b10aa5871da402c4a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:59:28.559311 18877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850e0887d8094dfc8c6d336ee961c1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:28.559314 19011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850e0887d8094dfc8c6d336ee961c1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:28.559182 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 850e0887d8094dfc8c6d336ee961c1cb (DEFAULT_TABLE table=test-table [id=ced59ba780eb4b10aa5871da402c4a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:59:28.560058 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850e0887d8094dfc8c6d336ee961c1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:28.567829 19096 tablet_bootstrap.cc:492] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: Bootstrap starting.
I20260620 14:59:28.567966 19097 tablet_bootstrap.cc:492] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Bootstrap starting.
I20260620 14:59:28.570267 19096 tablet_bootstrap.cc:654] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:28.570694 19097 tablet_bootstrap.cc:654] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:28.571182 19096 log.cc:824] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:28.571413 19097 log.cc:824] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:28.571904 19095 tablet_bootstrap.cc:492] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239: Bootstrap starting.
I20260620 14:59:28.573472 19096 tablet_bootstrap.cc:492] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: No bootstrap required, opened a new log
I20260620 14:59:28.573678 19097 tablet_bootstrap.cc:492] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: No bootstrap required, opened a new log
I20260620 14:59:28.573705 19096 ts_tablet_manager.cc:1397] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:59:28.573930 19097 ts_tablet_manager.cc:1397] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:59:28.574069 19095 tablet_bootstrap.cc:654] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:28.574936 19095 log.cc:824] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:28.577386 19095 tablet_bootstrap.cc:492] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239: No bootstrap required, opened a new log
I20260620 14:59:28.577574 19095 ts_tablet_manager.cc:1397] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:59:28.577883 19097 raft_consensus.cc:348] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.578147 19097 raft_consensus.cc:374] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:28.578197 19097 raft_consensus.cc:729] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2570dd4de07948449ae4515edb8e89a8, State: Initialized, Role: FOLLOWER
I20260620 14:59:28.578715 19097 consensus_queue.cc:260] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.578858 19096 raft_consensus.cc:348] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.579082 19096 raft_consensus.cc:374] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:28.579119 19096 raft_consensus.cc:729] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3431696f380e45e880d7ac7f768e9fc8, State: Initialized, Role: FOLLOWER
I20260620 14:59:28.579519 19096 consensus_queue.cc:260] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.579691 19097 ts_tablet_manager.cc:1428] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:59:28.580060 19077 heartbeater.cc:503] Master 127.7.46.190:34583 was elected leader, sending a full tablet report...
I20260620 14:59:28.580385 19096 ts_tablet_manager.cc:1428] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:59:28.580688 18943 heartbeater.cc:503] Master 127.7.46.190:34583 was elected leader, sending a full tablet report...
I20260620 14:59:28.581743 19095 raft_consensus.cc:348] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.581957 19095 raft_consensus.cc:374] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:28.582006 19095 raft_consensus.cc:729] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffcf72ed5b8b4fbd9aacb2a7297d8239, State: Initialized, Role: FOLLOWER
I20260620 14:59:28.582729 19095 consensus_queue.cc:260] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.583678 19095 ts_tablet_manager.cc:1428] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:59:28.584033 18805 heartbeater.cc:503] Master 127.7.46.190:34583 was elected leader, sending a full tablet report...
W20260620 14:59:28.608332 18806 tablet.cc:2369] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:59:28.641592 18944 tablet.cc:2369] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:59:28 dist-test-slave-30t4 krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
W20260620 14:59:28.742352 19078 tablet.cc:2369] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:59:28.905328 19101 raft_consensus.cc:482] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:59:28.905553 19101 raft_consensus.cc:504] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.906956 19101 leader_election.cc:288] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3431696f380e45e880d7ac7f768e9fc8 (127.7.46.130:35253), ffcf72ed5b8b4fbd9aacb2a7297d8239 (127.7.46.129:38051)
I20260620 14:59:28.911825 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.907193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:35253 (local address 127.7.46.131:58429)
0620 14:59:28.907324 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.907340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.907454 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.907961 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.907964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.907983 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:28.908104 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.908113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.909147 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.909150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.910417 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.910431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.911349 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.911352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.911499 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.911537 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.911597 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.911671 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":45}
I20260620 14:59:28.911825 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.907852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:34001 (local address 127.7.46.129:38051)
0620 14:59:28.907977 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.907981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.907993 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:59:28.908038 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.908041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.908088 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.908160 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:28.908490 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.909216 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.910567 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.911153 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.911239 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.911289 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.911636 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.911703 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:59:28.912247 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.907336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:38051 (local address 127.7.46.131:34001)
0620 14:59:28.907671 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.907685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.907816 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.908261 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.908263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.908279 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:28.908368 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.908374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.909333 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.909336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.910438 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.910444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.911367 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.911370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.911475 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.911520 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.912022 (+ 502us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.912106 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":9,"thread_start_us":123,"threads_started":1}
I20260620 14:59:28.912640 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.907456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:58429 (local address 127.7.46.130:35253)
0620 14:59:28.907623 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.907628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.907641 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:59:28.907692 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.907695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.907747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.907837 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:28.908227 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.909030 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.910566 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.911147 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.911249 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.911289 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.912439 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.912529 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:59:28.913137 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850e0887d8094dfc8c6d336ee961c1cb" candidate_uuid: "2570dd4de07948449ae4515edb8e89a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" is_pre_election: true
I20260620 14:59:28.913137 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850e0887d8094dfc8c6d336ee961c1cb" candidate_uuid: "2570dd4de07948449ae4515edb8e89a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3431696f380e45e880d7ac7f768e9fc8" is_pre_election: true
I20260620 14:59:28.913430 18759 raft_consensus.cc:2454] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2570dd4de07948449ae4515edb8e89a8 in term 0.
I20260620 14:59:28.913431 18897 raft_consensus.cc:2454] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2570dd4de07948449ae4515edb8e89a8 in term 0.
I20260620 14:59:28.913911 18966 leader_election.cc:302] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2570dd4de07948449ae4515edb8e89a8, 3431696f380e45e880d7ac7f768e9fc8; no voters:
I20260620 14:59:28.914234 19101 raft_consensus.cc:2781] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:59:28.914314 19101 raft_consensus.cc:482] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:59:28.914358 19101 raft_consensus.cc:3037] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:28.915661 19101 raft_consensus.cc:504] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.916062 19101 leader_election.cc:288] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [CANDIDATE]: Term 1 election: Requested vote from peers 3431696f380e45e880d7ac7f768e9fc8 (127.7.46.130:35253), ffcf72ed5b8b4fbd9aacb2a7297d8239 (127.7.46.129:38051)
I20260620 14:59:28.916397 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850e0887d8094dfc8c6d336ee961c1cb" candidate_uuid: "2570dd4de07948449ae4515edb8e89a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3431696f380e45e880d7ac7f768e9fc8"
I20260620 14:59:28.916443 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850e0887d8094dfc8c6d336ee961c1cb" candidate_uuid: "2570dd4de07948449ae4515edb8e89a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239"
I20260620 14:59:28.916518 18897 raft_consensus.cc:3037] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:28.916543 18759 raft_consensus.cc:3037] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:28.917681 18759 raft_consensus.cc:2454] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2570dd4de07948449ae4515edb8e89a8 in term 1.
I20260620 14:59:28.917681 18897 raft_consensus.cc:2454] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2570dd4de07948449ae4515edb8e89a8 in term 1.
I20260620 14:59:28.918018 18964 leader_election.cc:302] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2570dd4de07948449ae4515edb8e89a8, ffcf72ed5b8b4fbd9aacb2a7297d8239; no voters:
I20260620 14:59:28.918217 19101 raft_consensus.cc:2781] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:28.918478 19101 raft_consensus.cc:686] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 1 LEADER]: Becoming Leader. State: Replica: 2570dd4de07948449ae4515edb8e89a8, State: Running, Role: LEADER
I20260620 14:59:28.918816 19101 consensus_queue.cc:237] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } }
I20260620 14:59:28.922217 18591 catalog_manager.cc:5495] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2570dd4de07948449ae4515edb8e89a8 (127.7.46.131). New cstate: current_term: 1 leader_uuid: "2570dd4de07948449ae4515edb8e89a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2570dd4de07948449ae4515edb8e89a8" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 44683 } health_report { overall_health: HEALTHY } } }
I20260620 14:59:29.011965 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.007859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.7.46.131:44683)
0620 14:59:29.008007 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.008012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.008026 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:59:29.008111 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.008114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.008177 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.008288 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:29.008619 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.009393 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.010287 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.010627 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.010734 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.010992 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:29.011531 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:29.011651 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.011729 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.011810 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:59:29.019042 18759 raft_consensus.cc:1261] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 1 FOLLOWER]: Refusing update from remote peer 2570dd4de07948449ae4515edb8e89a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:59:29.019029 18897 raft_consensus.cc:1261] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 1 FOLLOWER]: Refusing update from remote peer 2570dd4de07948449ae4515edb8e89a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:59:29.019867 19101 consensus_queue.cc:1035] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffcf72ed5b8b4fbd9aacb2a7297d8239" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:59:29.020081 19107 consensus_queue.cc:1035] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3431696f380e45e880d7ac7f768e9fc8" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:59:29.027884 19117 mvcc.cc:205] Tried to move back new op lower bound from 7298939162694279168 to 7298939162294538240. Current Snapshot: MvccSnapshot[applied={T|T < 7298939162694279168}]
I20260620 14:59:29.031462 19118 mvcc.cc:205] Tried to move back new op lower bound from 7298939162694279168 to 7298939162294538240. Current Snapshot: MvccSnapshot[applied={T|T < 7298939162694279168}]
I20260620 14:59:29.034318 19119 mvcc.cc:205] Tried to move back new op lower bound from 7298939162694279168 to 7298939162294538240. Current Snapshot: MvccSnapshot[applied={T|T < 7298939162694279168}]
I20260620 14:59:29.037940 18593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:59:29.038113 18593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:59:29.040691 18593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61a47262c734e3583b70018ab82aa45: Sending DeleteTablet for 3 replicas of tablet 850e0887d8094dfc8c6d336ee961c1cb
I20260620 14:59:29.041404 18877 tablet_service.cc:1514] Processing DeleteTablet for tablet 850e0887d8094dfc8c6d336ee961c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:59:29 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:58468
I20260620 14:59:29.041733 19011 tablet_service.cc:1514] Processing DeleteTablet for tablet 850e0887d8094dfc8c6d336ee961c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:59:29 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:52440
I20260620 14:59:29.041872 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 850e0887d8094dfc8c6d336ee961c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:59:29 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:48320
I20260620 14:59:29.042356 19128 tablet_replica.cc:330] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239: stopping tablet replica
I20260620 14:59:29.042748 19128 raft_consensus.cc:2229] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:59:29.043082 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 18674
I20260620 14:59:29.043167 19128 raft_consensus.cc:2258] T 850e0887d8094dfc8c6d336ee961c1cb P ffcf72ed5b8b4fbd9aacb2a7297d8239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:59:29.048024 19126 tablet_replica.cc:330] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: stopping tablet replica
I20260620 14:59:29.048374 19126 raft_consensus.cc:2229] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:59:29.048693 19126 raft_consensus.cc:2258] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:59:29.050709 19126 ts_tablet_manager.cc:1905] T 850e0887d8094dfc8c6d336ee961c1cb P 3431696f380e45e880d7ac7f768e9fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:59:29.051956 19127 tablet_replica.cc:330] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: stopping tablet replica
W20260620 14:59:29.051970 18577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:38051 (error 108)
I20260620 14:59:29.052225 19127 raft_consensus.cc:2229] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:59:29.052541 19127 raft_consensus.cc:2258] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:59:29.053040 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 18813
I20260620 14:59:29.054549 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.054211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:38051 (local address 127.0.0.1:48328)
0620 14:59:29.054396 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:29.054463 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:38051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260620 14:59:29.054786 18577 catalog_manager.cc:4584] TS ffcf72ed5b8b4fbd9aacb2a7297d8239 (127.7.46.129:38051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 850e0887d8094dfc8c6d336ee961c1cb: Network error: Client connection negotiation failed: client connection to 127.7.46.129:38051: connect: Connection refused (error 111)
I20260620 14:59:29.054872 19127 ts_tablet_manager.cc:1905] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:59:29.057385 19127 ts_tablet_manager.cc:1918] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:59:29.057505 19127 log.cc:1196] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/850e0887d8094dfc8c6d336ee961c1cb
I20260620 14:59:29.057823 19127 ts_tablet_manager.cc:1939] T 850e0887d8094dfc8c6d336ee961c1cb P 2570dd4de07948449ae4515edb8e89a8: Deleting consensus metadata
I20260620 14:59:29.058996 18577 catalog_manager.cc:4841] TS 2570dd4de07948449ae4515edb8e89a8 (127.7.46.131:44683): tablet 850e0887d8094dfc8c6d336ee961c1cb (table test-table [id=ced59ba780eb4b10aa5871da402c4a72]) successfully deleted
I20260620 14:59:29.060326 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 18948
I20260620 14:59:29.062433 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.062040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:35253 (local address 127.0.0.1:58478)
0620 14:59:29.062239 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:29.062325 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.130:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 14:59:29.066434 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 18561
I20260620 14:59:29.073081 7354 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:59:29.417834 7354 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:59:29.417964 7354 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:59:29.762372 7354 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:59:29.762 UTC [18147] LOG: received smart shutdown request
2026-06-20 14:59:29.765 UTC [18147] LOG: background worker "logical replication launcher" (PID 18156) exited with exit code 1
2026-06-20 14:59:29.766 UTC [18151] LOG: shutting down
2026-06-20 14:59:29.905 UTC [18147] LOG: database system is shut down
2026-06-20T14:59:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45133 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:30 dist-test-slave-30t4 krb5kdc[19138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:30 dist-test-slave-30t4 krb5kdc[19138](info): set up 2 sockets
Jun 20 14:59:30 dist-test-slave-30t4 krb5kdc[19138](info): commencing operation
krb5kdc: starting...
W20260620 14:59:31.051935 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 14:59:31 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.46.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.46.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/rangeradmin_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/rangeradmin_127.7.46.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.46.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.46.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/rangerlookup_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/rangerlookup_127.7.46.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.46.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.46.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/HTTP_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/HTTP_127.7.46.148@KRBTEST.COM.keytab.
I20260620 14:59:31.210549 7354 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLLAc44/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/postgres -l logfile start
2026-06-20 14:59:33.590 UTC [19168] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:59:33.590 UTC [19168] LOG: listening on IPv4 address "127.7.46.148", port 59355
2026-06-20 14:59:33.593 UTC [19168] LOG: listening on Unix socket "/tmp/.s.PGSQL.59355"
2026-06-20 14:59:33.613 UTC [19171] LOG: database system was shut down at 2026-06-20 14:59:32 UTC
2026-06-20 14:59:33.616 UTC [19168] LOG: database system is ready to accept connections
I20260620 14:59:34.663596 7354 mini_postgres.cc:96] Postgres bound to 59355
2026-06-20 14:59:34.669 UTC [19179] FATAL: database "slave" does not exist
127.7.46.148:59355 - accepting connections
I20260620 14:59:34.670140 7354 mini_ranger.cc:161] Starting Ranger...
I20260620 14:59:34.684864 7354 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:59:35.123456 7354 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:59:35.123554 7354 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-admin
2026-06-20 14:59:35,386 [I] DB FLAVOR :POSTGRES
2026-06-20 14:59:35,387 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:59:35,387 [I] Checking connection..
2026-06-20 14:59:35,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:59:35,640 [I] Checking connection passed.
2026-06-20 14:59:35,640 [I] --------- Verifying version history table ---------
2026-06-20 14:59:35,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:59:35,894 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:59:35,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:59:36,118 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:59:36,118 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:59:36,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:59:36.353 UTC [19243] WARNING: there is no transaction in progress
2026-06-20 14:59:36,359 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:59:36,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:59:36,612 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:59:36,612 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:59:36,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:59:36,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:59:37,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-30t4.c.gcp-upstream.internal','N') ;"
2026-06-20 14:59:37,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:59:37,608 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:59:37,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:59:37,860 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:59:37,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:59:38,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:59:38,310 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:59:38,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:59:39.208 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.223 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.239 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.253 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.268 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.284 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.345 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.351 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.365 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.375 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.385 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.398 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.407 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.412 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.418 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:39.424 UTC [19351] WARNING: there is no transaction in progress
2026-06-20 14:59:40,034 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:59:40,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:59:40,272 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:59:40,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:59:40,498 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:59:40,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:59:40,725 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:59:40,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:59:40,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:59:41,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-30t4.c.gcp-upstream.internal';"
2026-06-20 14:59:41,451 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:59:41,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:59:41,705 [I] DB_PATCHES have already been applied
I20260620 14:59:41.710749 7354 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLLAc44/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:59:41.710816 7354 mini_ranger.cc:193] Using host: 127.7.46.148
I20260620 14:59:41.714161 7354 mini_ranger.cc:227] Ranger admin URL: http://127.7.46.148:53499
Jun 20, 2026 2:59:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:59:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:59:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:59:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:59:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:59:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:59:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:59:42.806370 7354 mini_ranger.cc:160] Time spent starting Ranger: real 8.136s user 0.005s sys 0.005s
Jun 20, 2026 3:00:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:00:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 3:00:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:00:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:00:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:00:04.095286 7354 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.46.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.46.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/rangerkms_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.46.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/rangerkms_127.7.46.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.46.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.46.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.46.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/HTTP_127.7.46.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.46.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/HTTP_127.7.46.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:00:04 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967604, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:00:04.180721 7354 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:00:04.190621 7354 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:00:05.042047 7354 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:00:05.042141 7354 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms
I20260620 15:00:05.042164 7354 mini_ranger_kms.cc:227] Using postgres at 127.7.46.148:59355
2026-06-20 15:00:05,208 [I] DB FLAVOR :POSTGRES
2026-06-20 15:00:05,208 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:00:05,208 [I] Checking connection
2026-06-20 15:00:05,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:00:05,432 [I] connection success
2026-06-20 15:00:05,433 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:00:05,433 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:00:05,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:00:05,660 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:00:05,660 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:00:05,660 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:00:05,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.46.148:59355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:00:05,918 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:00:06.346539 7354 mini_ranger_kms.cc:310] Created kms service
I20260620 15:00:06.531839 7354 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:00:06.576264 7354 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:00:06 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967604, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:06.960493 7354 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:00:06.960891 7354 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLLAc44/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLLAc44/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:00:06.960944 7354 mini_ranger_kms.cc:240] Using host: 127.7.46.148
I20260620 15:00:06.965013 7354 mini_ranger_kms.cc:276] Ranger KMS PID: 19531
I20260620 15:00:06.965102 7354 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.46.148:41347
Jun 20, 2026 3:00:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:00:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:00:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:00:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:00:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:00:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 15:00:09.176930 7354 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.996s user 0.019s sys 0.131s
I20260620 15:00:09.177057 7354 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:00:09.177145 7354 mini_ranger_kms.cc:401] 127.7.46.148:41347/kms/v1/keys
Jun 20 15:00:09 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967609, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.46.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 15:00:09 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967609, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.46.148@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
Jun 20 15:00:10 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967609, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.46.148@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
Jun 20 15:00:12 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:00:12.236943 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:33343
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:33343
--ranger_config_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:41347/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:12.343712 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:12.344013 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:12.344080 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:12.348012 19582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:12.348093 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:12.348138 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:12.348191 19582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:12.348232 19582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:12.353535 19582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.46.148:41347/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:33343
--ranger_config_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:12.354585 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:12.355584 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:12.364203 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:12.364218 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:12.364262 19582 server_base.cc:1034] running on GCE node
W20260620 15:00:12.364213 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:12.364917 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:12.366070 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:12.367233 19582 hybrid_clock.cc:648] HybridClock initialized: now 1781967612367220 us; error 26 us; skew 500 ppm
Jun 20 15:00:12 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967612, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:12.370025 19582 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:00:12.370848 19582 webserver.cc:460] Webserver started at http://127.7.46.190:41513/ using document root <none> and password file <none>
I20260620 15:00:12.371384 19582 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:12.371443 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:12.371668 19582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:00:12 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967612, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:12.416894 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "41282de409a44460aaf6a0dbae190be3"
format_stamp: "Formatted at 2026-06-20 15:00:12 on dist-test-slave-30t4"
server_key: "72cc0587fda0c8a5ea35cc1f873cd113"
server_key_iv: "098fa4a900a167861cb1da6a3318473a"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:12.417598 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "41282de409a44460aaf6a0dbae190be3"
format_stamp: "Formatted at 2026-06-20 15:00:12 on dist-test-slave-30t4"
server_key: "72cc0587fda0c8a5ea35cc1f873cd113"
server_key_iv: "098fa4a900a167861cb1da6a3318473a"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:12.421690 19582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:00:12.435479 19597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:12.436950 19582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:12.437117 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "41282de409a44460aaf6a0dbae190be3"
format_stamp: "Formatted at 2026-06-20 15:00:12 on dist-test-slave-30t4"
server_key: "72cc0587fda0c8a5ea35cc1f873cd113"
server_key_iv: "098fa4a900a167861cb1da6a3318473a"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:12.437266 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:12.449446 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:12.452452 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:12.452666 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:12.460323 19582 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:33343
I20260620 15:00:12.460347 19649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:33343 every 8 connection(s)
I20260620 15:00:12.461302 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:12.463531 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 19582
I20260620 15:00:12.463665 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:00:12.464004 7354 external_mini_cluster.cc:1428] Setting key 58e62fadd78ae28fc01fe635ad16fb39
I20260620 15:00:12.464762 19650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:12.471803 19650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Bootstrap starting.
Jun 20 15:00:12 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:12.474613 19650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:12.476032 19650 log.cc:824] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:12.478039 19650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: No bootstrap required, opened a new log
I20260620 15:00:12.478045 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:12.465447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.7.46.190:33343)
0620 15:00:12.465863 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:12.465882 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:12.465924 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:00:12.466565 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:12.466580 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:12.466914 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:12.467242 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:12.468089 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:12.469055 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:12.469885 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:12.470354 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:12.470539 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:12.473500 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:12.473505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:12.473524 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:12.473550 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:12.475300 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:12.475730 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:12.475736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:12.475742 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:12.475910 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:12.476199 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:12.476202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:12.476204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:12.476597 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:12.476793 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:12.477082 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:00:12.477358 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":136,"threads_started":1}
I20260620 15:00:12.481405 19650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } }
I20260620 15:00:12.481644 19650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:12.481735 19650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41282de409a44460aaf6a0dbae190be3, State: Initialized, Role: FOLLOWER
I20260620 15:00:12.482184 19650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } }
I20260620 15:00:12.482331 19650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:12.482415 19650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:12.482510 19650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:12.483585 19650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } }
I20260620 15:00:12.483950 19650 leader_election.cc:302] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41282de409a44460aaf6a0dbae190be3; no voters:
I20260620 15:00:12.484311 19650 leader_election.cc:288] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:12.484452 19655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:12.484715 19655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 1 LEADER]: Becoming Leader. State: Replica: 41282de409a44460aaf6a0dbae190be3, State: Running, Role: LEADER
I20260620 15:00:12.485067 19655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } }
I20260620 15:00:12.485525 19650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:12.486761 19657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41282de409a44460aaf6a0dbae190be3. Latest consensus state: current_term: 1 leader_uuid: "41282de409a44460aaf6a0dbae190be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } } }
I20260620 15:00:12.486794 19656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41282de409a44460aaf6a0dbae190be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } } }
I20260620 15:00:12.486876 19657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:12.486883 19656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:12.487223 19664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:12.490327 19664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:12.490720 19650 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:00:12.499339 19664 catalog_manager.cc:1347] Generated new cluster ID: 5f034fadea8340e7afa32d04eaa55ce0
I20260620 15:00:12.499460 19664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:12.510095 19664 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:12.511360 19664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:12.518550 19664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Generated new TSK 0
I20260620 15:00:12.519352 19664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 15:00:13.183259 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:41347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33343
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:13.320261 19695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:13.320513 19695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:13.320640 19695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:13.324658 19695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:13.324733 19695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:13.324870 19695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:00:13.331020 19695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.46.148:41347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:13.332106 19695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:13.333133 19695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:13.342657 19702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:13.342778 19705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:13.343174 19703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:13.343413 19695 server_base.cc:1034] running on GCE node
I20260620 15:00:13.343843 19695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:13.344487 19695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:13.345715 19695 hybrid_clock.cc:648] HybridClock initialized: now 1781967613345695 us; error 37 us; skew 500 ppm
Jun 20 15:00:13 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:13.348534 19695 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:00:13.349395 19695 webserver.cc:460] Webserver started at http://127.7.46.129:42143/ using document root <none> and password file <none>
I20260620 15:00:13.349983 19695 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:13.350042 19695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:13.350253 19695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:00:13 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:13.365634 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "3cda9039ef5d45b6ae6545e2f8342e1a"
format_stamp: "Formatted at 2026-06-20 15:00:13 on dist-test-slave-30t4"
server_key: "8228435fd407e06ab6bbf5abf02b6e06"
server_key_iv: "c97b10110d50c300a18fcef97ea785df"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:13.366277 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "3cda9039ef5d45b6ae6545e2f8342e1a"
format_stamp: "Formatted at 2026-06-20 15:00:13 on dist-test-slave-30t4"
server_key: "8228435fd407e06ab6bbf5abf02b6e06"
server_key_iv: "c97b10110d50c300a18fcef97ea785df"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:13.373193 19695 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 15:00:13.396234 19712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:13.397974 19695 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260620 15:00:13.398110 19695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "3cda9039ef5d45b6ae6545e2f8342e1a"
format_stamp: "Formatted at 2026-06-20 15:00:13 on dist-test-slave-30t4"
server_key: "8228435fd407e06ab6bbf5abf02b6e06"
server_key_iv: "c97b10110d50c300a18fcef97ea785df"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:13.398250 19695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:13.447576 19695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:13.451296 19695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:13.451494 19695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:13.452253 19695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:13.453266 19695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:13.453327 19695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:13.453382 19695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:13.453459 19695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 15:00:13.463697 19695 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:41613
I20260620 15:00:13.463795 19827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:41613 every 8 connection(s)
I20260620 15:00:13.464807 19695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:13.471719 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 19695
I20260620 15:00:13.471871 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:13.472184 7354 external_mini_cluster.cc:1428] Setting key a8026975fe2dca409c91df81da01442c
Jun 20 15:00:13 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:13.481921 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:13.467351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33343 (local address 127.7.46.129:44961)
0620 15:00:13.468221 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:13.468272 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:13.469132 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:13.469700 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:13.469707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:13.469997 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:13.470455 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:13.470471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:13.471281 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:13.471285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:13.472649 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:13.472660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:13.473359 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:13.473364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:13.473569 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:13.474261 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:13.474282 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:13.476002 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:13.478411 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:13.478419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:13.478436 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:13.478804 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:13.479155 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:13.479161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:13.479164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:13.479352 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:13.479913 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:13.479922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:13.480260 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 15:00:13.480492 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 15:00:13.480851 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":140,"threads_started":1}
I20260620 15:00:13.482975 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:13.467639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:44961 (local address 127.7.46.190:33343)
0620 15:00:13.467953 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:13.467971 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:13.468390 (+ 419us) server_negotiation.cc:407] Connection header received
0620 15:00:13.469376 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:13.469380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:13.469437 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:13.469548 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:13.470599 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:13.471155 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:13.472788 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:13.473093 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:13.473168 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:13.476158 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:13.476165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:13.476171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:13.476205 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:13.478279 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:13.478927 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:13.478934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:13.478938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:13.479041 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:13.479485 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:13.479490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:13.479493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:13.479725 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:13.479855 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:13.482639 (+ 2784us) server_negotiation.cc:299] Negotiation successful
0620 15:00:13.482771 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 15:00:13.484747 19828 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33343
I20260620 15:00:13.484989 19828 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:13.485517 19828 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:13.487205 19614 ts_manager.cc:194] Registered new tserver with Master: 3cda9039ef5d45b6ae6545e2f8342e1a (127.7.46.129:41613)
Jun 20 15:00:13 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 15:00:13.488461 19614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:44961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 15:00:13.505112 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:41347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33343
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:13.657140 19834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:13.657555 19834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:13.657682 19834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:13.663594 19834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:13.663726 19834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:13.663874 19834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:00:13.671579 19834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.46.148:41347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:13.672915 19834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:13.674268 19834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:13.686301 19843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:13.686545 19841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:13.687855 19834 server_base.cc:1034] running on GCE node
W20260620 15:00:13.691924 19840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:13.692444 19834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:13.693199 19834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:13.694396 19834 hybrid_clock.cc:648] HybridClock initialized: now 1781967613694354 us; error 50 us; skew 500 ppm
Jun 20 15:00:13 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:13.697929 19834 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:00:13.699050 19834 webserver.cc:460] Webserver started at http://127.7.46.130:41739/ using document root <none> and password file <none>
I20260620 15:00:13.699819 19834 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:13.699976 19834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:13.700278 19834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:00:13 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:13.721364 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "6576c4925ce443b2aa91d908ecbcb568"
format_stamp: "Formatted at 2026-06-20 15:00:13 on dist-test-slave-30t4"
server_key: "f69cf3bf6044b5f77c37b3e8ba98cb48"
server_key_iv: "f98239530e691c880a8c583f99b47876"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:13.721966 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "6576c4925ce443b2aa91d908ecbcb568"
format_stamp: "Formatted at 2026-06-20 15:00:13 on dist-test-slave-30t4"
server_key: "f69cf3bf6044b5f77c37b3e8ba98cb48"
server_key_iv: "f98239530e691c880a8c583f99b47876"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:13.725870 19834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:00:13.738991 19850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:13.746578 19834 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.002s
I20260620 15:00:13.746727 19834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "6576c4925ce443b2aa91d908ecbcb568"
format_stamp: "Formatted at 2026-06-20 15:00:13 on dist-test-slave-30t4"
server_key: "f69cf3bf6044b5f77c37b3e8ba98cb48"
server_key_iv: "f98239530e691c880a8c583f99b47876"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:13.746866 19834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:13.781122 19834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:13.784821 19834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:13.785029 19834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:13.785708 19834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:13.786751 19834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:13.786811 19834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:13.786866 19834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:13.786949 19834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:13.798394 19834 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:46737
I20260620 15:00:13.798465 19963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:46737 every 8 connection(s)
I20260620 15:00:13.799540 19834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:13.803634 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 19834
I20260620 15:00:13.804960 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:13.805318 7354 external_mini_cluster.cc:1428] Setting key dcb6d9954a6e9fdd561d99c290b2e162
Jun 20 15:00:13 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:13.816550 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:13.801543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:37425 (local address 127.7.46.190:33343)
0620 15:00:13.801677 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:13.801682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:13.803996 (+ 2314us) server_negotiation.cc:407] Connection header received
0620 15:00:13.804588 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:13.804592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:13.804650 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:13.804705 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:13.806300 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:13.807110 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:13.808809 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:13.809170 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:13.809250 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:13.812221 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:13.812225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:13.812228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:13.812254 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:13.814081 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:13.814797 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:13.814800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:13.814802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:13.814886 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:13.815277 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:13.815281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:13.815283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:13.815491 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:13.815587 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:13.816249 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 15:00:13.816373 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:00:13.818030 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:13.802034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33343 (local address 127.7.46.130:37425)
0620 15:00:13.803191 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:13.803247 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:13.804324 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:13.805118 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:13.805127 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:13.805477 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:13.806128 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:13.806146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:13.807268 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:13.807271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:13.808668 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:13.808678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:13.809443 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:13.809447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:13.809635 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:13.810303 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:13.810325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:13.812034 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:13.814218 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:13.814224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:13.814238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:13.814673 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:13.814996 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:13.814999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:13.815001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:13.815166 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:13.815654 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:13.815660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:13.815979 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 15:00:13.816755 (+ 776us) client_negotiation.cc:240] Negotiation successful
0620 15:00:13.817066 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":133,"threads_started":1}
I20260620 15:00:13.819267 19964 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33343
I20260620 15:00:13.819494 19964 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:13.820050 19964 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:13.821099 19614 ts_manager.cc:194] Registered new tserver with Master: 6576c4925ce443b2aa91d908ecbcb568 (127.7.46.130:46737)
I20260620 15:00:13.821830 19614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:37425
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 15:00:13.849220 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:41347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33343
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:00:13 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
W20260620 15:00:13.993024 19969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:13.993381 19969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:13.993444 19969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:13.998883 19969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:13.998968 19969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:13.999050 19969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:00:14.005870 19969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.46.148:41347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:14.006909 19969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:14.008006 19969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:14.018131 19974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:14.018589 19977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:14.018672 19975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:14.018813 19969 server_base.cc:1034] running on GCE node
I20260620 15:00:14.019222 19969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:14.019953 19969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:14.021165 19969 hybrid_clock.cc:648] HybridClock initialized: now 1781967614021145 us; error 34 us; skew 500 ppm
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:14.024601 19969 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:00:14.025640 19969 webserver.cc:460] Webserver started at http://127.7.46.131:46721/ using document root <none> and password file <none>
I20260620 15:00:14.026319 19969 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:14.026407 19969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:14.026664 19969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:14.044572 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "d65d90af66e14d02af547a72bdd17d3b"
format_stamp: "Formatted at 2026-06-20 15:00:14 on dist-test-slave-30t4"
server_key: "f26ccbef23f5c04ce480b900120f41da"
server_key_iv: "a0ae16ddc32c567d83756884ed761b69"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:14.046571 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "d65d90af66e14d02af547a72bdd17d3b"
format_stamp: "Formatted at 2026-06-20 15:00:14 on dist-test-slave-30t4"
server_key: "f26ccbef23f5c04ce480b900120f41da"
server_key_iv: "a0ae16ddc32c567d83756884ed761b69"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:14.053344 19969 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 15:00:14.072170 19984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.073972 19969 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 15:00:14.074103 19969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "d65d90af66e14d02af547a72bdd17d3b"
format_stamp: "Formatted at 2026-06-20 15:00:14 on dist-test-slave-30t4"
server_key: "f26ccbef23f5c04ce480b900120f41da"
server_key_iv: "a0ae16ddc32c567d83756884ed761b69"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:14.074234 19969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:14.091259 19969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:14.094592 19969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:14.094764 19969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:14.095412 19969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:14.096352 19969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:14.096417 19969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.096464 19969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:14.096491 19969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.105885 19969 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:33515
I20260620 15:00:14.106106 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:33515 every 8 connection(s)
I20260620 15:00:14.106952 19969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:14.110193 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 19969
I20260620 15:00:14.110318 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:14.110636 7354 external_mini_cluster.cc:1428] Setting key d846e1c509dfea66ceaa932a38256bf0
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:14.126525 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:14.113057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:59079 (local address 127.7.46.190:33343)
0620 15:00:14.113179 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:14.113184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:14.113743 (+ 559us) server_negotiation.cc:407] Connection header received
0620 15:00:14.114877 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:14.114880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:14.114933 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:14.115021 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:14.116451 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.117266 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.118915 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.119257 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.119360 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:14.122329 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:14.122333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:14.122336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:14.122363 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:14.124187 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:14.124807 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:14.124810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:14.124812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:14.124898 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:14.125308 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:14.125312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:14.125313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:14.125529 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:14.125615 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:14.126229 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 15:00:14.126354 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:00:14.128355 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:14.112005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33343 (local address 127.7.46.131:59079)
0620 15:00:14.113557 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:14.113629 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:14.114644 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:14.115217 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:14.115227 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:14.115589 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:14.116285 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:14.116305 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.117412 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.117415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:14.118780 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:14.118791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.119533 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.119538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:14.119730 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:14.120481 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:14.120501 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:14.122157 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:14.124320 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:14.124326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:14.124341 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:14.124684 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:14.125000 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:14.125003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:14.125005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:14.125182 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:14.125666 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:14.125677 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:14.125971 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 15:00:14.126963 (+ 992us) client_negotiation.cc:240] Negotiation successful
0620 15:00:14.127290 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"thread_start_us":132,"threads_started":1}
I20260620 15:00:14.129791 20098 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33343
I20260620 15:00:14.130050 20098 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:14.130640 20098 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:14.131717 19614 ts_manager.cc:194] Registered new tserver with Master: d65d90af66e14d02af547a72bdd17d3b (127.7.46.131:33515)
I20260620 15:00:14.132464 19614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:59079
I20260620 15:00:14.136193 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:14.136901 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 19695
I20260620 15:00:14.143090 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 19834
I20260620 15:00:14.149722 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 19969
I20260620 15:00:14.157455 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 19582
I20260620 15:00:14.165198 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:33343
--webserver_interface=127.7.46.190
--webserver_port=41513
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:33343
--ranger_config_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:41347/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.46.148:41347/kms with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:14.271045 20101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:14.271281 20101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:14.271353 20101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:14.275290 20101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:14.275345 20101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:14.275364 20101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:14.275380 20101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:14.275394 20101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:14.280817 20101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.46.148:41347/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:33343
--ranger_config_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=41513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:14.281754 20101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:14.282706 20101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:14.290968 20109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:14.290982 20106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:14.291088 20101 server_base.cc:1034] running on GCE node
W20260620 15:00:14.290987 20107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:14.291819 20101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:14.292851 20101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:14.294020 20101 hybrid_clock.cc:648] HybridClock initialized: now 1781967614294007 us; error 27 us; skew 500 ppm
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:14.296685 20101 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:00:14.297516 20101 webserver.cc:460] Webserver started at http://127.7.46.190:41513/ using document root <none> and password file <none>
I20260620 15:00:14.297988 20101 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:14.298074 20101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:14.347806 20101 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:14.350020 20117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.351411 20101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:14.351540 20101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "41282de409a44460aaf6a0dbae190be3"
format_stamp: "Formatted at 2026-06-20 15:00:12 on dist-test-slave-30t4"
server_key: "72cc0587fda0c8a5ea35cc1f873cd113"
server_key_iv: "098fa4a900a167861cb1da6a3318473a"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:14.352144 20101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:14.363631 20101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:14.366528 20101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:14.366725 20101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:14.373692 20101 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:33343
I20260620 15:00:14.373711 20169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:33343 every 8 connection(s)
I20260620 15:00:14.374689 20101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:14.378798 20170 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:00:14.380379 20170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Bootstrap starting.
I20260620 15:00:14.381599 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 20101
I20260620 15:00:14.382058 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:41347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:41613
--local_ip_for_outbound_sockets=127.7.46.129
--tserver_master_addrs=127.7.46.190:33343
--webserver_port=42143
--webserver_interface=127.7.46.129
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.46.148:41347/kms with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:14.396315 20170 log.cc:824] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:14.402587 20170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:00:14.402966 20170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Bootstrap complete.
I20260620 15:00:14.406965 20170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } }
I20260620 15:00:14.407346 20170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41282de409a44460aaf6a0dbae190be3, State: Initialized, Role: FOLLOWER
I20260620 15:00:14.407861 20170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } }
I20260620 15:00:14.408001 20170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:14.408106 20170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:14.408221 20170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:00:14.409981 20170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } }
I20260620 15:00:14.410295 20170 leader_election.cc:302] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41282de409a44460aaf6a0dbae190be3; no voters:
I20260620 15:00:14.410614 20170 leader_election.cc:288] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:00:14.410691 20174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:00:14.410934 20174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [term 2 LEADER]: Becoming Leader. State: Replica: 41282de409a44460aaf6a0dbae190be3, State: Running, Role: LEADER
I20260620 15:00:14.411274 20174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } }
I20260620 15:00:14.411841 20170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:14.412214 20176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41282de409a44460aaf6a0dbae190be3. Latest consensus state: current_term: 2 leader_uuid: "41282de409a44460aaf6a0dbae190be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } } }
I20260620 15:00:14.412398 20176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:14.412586 20175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "41282de409a44460aaf6a0dbae190be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41282de409a44460aaf6a0dbae190be3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 33343 } } }
I20260620 15:00:14.412691 20175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:14.414124 20179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:14.414505 20170 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:00:14.417209 20179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:14.417821 20179 catalog_manager.cc:1259] Loaded cluster ID: 5f034fadea8340e7afa32d04eaa55ce0
I20260620 15:00:14.417886 20179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:14.423308 20179 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:14.424408 20179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Loaded TSK: 0
I20260620 15:00:14.425817 20179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:00:14.519656 20172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:14.520079 20172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:14.520201 20172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:14.526229 20172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:14.526348 20172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:14.526494 20172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:00:14.533941 20172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.46.148:41347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:41613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=42143
--tserver_master_addrs=127.7.46.190:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:14.535166 20172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:14.536398 20172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:14.548080 20210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:14.548059 20208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:14.548058 20207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:14.548800 20172 server_base.cc:1034] running on GCE node
I20260620 15:00:14.549227 20172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:14.549923 20172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:14.551131 20172 hybrid_clock.cc:648] HybridClock initialized: now 1781967614551112 us; error 34 us; skew 500 ppm
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:14.555415 20172 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:00:14.556488 20172 webserver.cc:460] Webserver started at http://127.7.46.129:42143/ using document root <none> and password file <none>
I20260620 15:00:14.557273 20172 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:14.557370 20172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:14.580163 20172 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:00:14.582707 20218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.584627 20172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:14.584793 20172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "3cda9039ef5d45b6ae6545e2f8342e1a"
format_stamp: "Formatted at 2026-06-20 15:00:13 on dist-test-slave-30t4"
server_key: "8228435fd407e06ab6bbf5abf02b6e06"
server_key_iv: "c97b10110d50c300a18fcef97ea785df"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:14.585446 20172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:14.615559 20172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:14.620198 20172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:14.620538 20172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:14.621272 20172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:14.622355 20172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:14.622457 20172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.622550 20172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:14.622606 20172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.633136 20172 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:41613
I20260620 15:00:14.633281 20331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:41613 every 8 connection(s)
I20260620 15:00:14.634246 20172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:14.640805 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 20172
I20260620 15:00:14.641335 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:41347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:46737
--local_ip_for_outbound_sockets=127.7.46.130
--tserver_master_addrs=127.7.46.190:33343
--webserver_port=41739
--webserver_interface=127.7.46.130
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.46.148:41347/kms with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:14.658974 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:14.636576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:37987 (local address 127.7.46.190:33343)
0620 15:00:14.638400 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:14.638415 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:14.638446 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:00:14.638660 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:14.638669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:14.638939 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:14.639240 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:14.640493 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.641985 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.643790 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.644575 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.644719 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:14.652884 (+ 8165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:14.652891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:14.652909 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:14.652941 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:14.655018 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:14.655683 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:14.655690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:14.655695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:14.655853 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:14.656484 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:14.656487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:14.656489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:14.656931 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:14.657043 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:14.657726 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 15:00:14.658008 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1696,"thread_start_us":111,"threads_started":1}
I20260620 15:00:14.660869 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:14.636713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33343 (local address 127.7.46.129:37987)
0620 15:00:14.637202 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:14.637252 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:14.638037 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:14.639301 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:14.639309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:14.639641 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:14.640310 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:14.640331 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.642139 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.642145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:14.643617 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:14.643630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.646162 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.646168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:14.646395 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:14.650706 (+ 4311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:14.650726 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:14.652641 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:14.655127 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:14.655134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:14.655149 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:14.655549 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:14.655969 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:14.655972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:14.655974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:14.656313 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:14.657125 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:14.657131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:14.657396 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 15:00:14.659653 (+ 2257us) client_negotiation.cc:240] Negotiation successful
0620 15:00:14.660021 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":109,"threads_started":1}
I20260620 15:00:14.663576 20332 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33343
I20260620 15:00:14.663859 20332 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:14.664358 20332 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
W20260620 15:00:14.666486 20332 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:33343 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:00:14.666613 20332 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:14.666782 20332 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:14.667524 20332 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:14.667701 20332 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
W20260620 15:00:14.668457 20332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:00:14.782395 20336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:14.782727 20336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:14.782824 20336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:14.787578 20336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:14.787665 20336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:14.787750 20336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:00:14.793458 20336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.46.148:41347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:46737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=41739
--tserver_master_addrs=127.7.46.190:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:14.794608 20336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:14.795707 20336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:14.806797 20344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:14.806958 20342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:14.807337 20336 server_base.cc:1034] running on GCE node
W20260620 15:00:14.807261 20341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:14.807976 20336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:14.808746 20336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:14.812042 20336 hybrid_clock.cc:648] HybridClock initialized: now 1781967614811988 us; error 63 us; skew 500 ppm
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:14.815627 20336 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:00:14.816728 20336 webserver.cc:460] Webserver started at http://127.7.46.130:41739/ using document root <none> and password file <none>
I20260620 15:00:14.817399 20336 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:14.817492 20336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:14.834451 20336 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:00:14.840590 20352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.841954 20336 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 15:00:14.842098 20336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "6576c4925ce443b2aa91d908ecbcb568"
format_stamp: "Formatted at 2026-06-20 15:00:13 on dist-test-slave-30t4"
server_key: "f69cf3bf6044b5f77c37b3e8ba98cb48"
server_key_iv: "f98239530e691c880a8c583f99b47876"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:14.842723 20336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:14.860972 20336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:14.864434 20336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:14.864631 20336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:14.865275 20336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:14.866247 20336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:14.866308 20336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.866361 20336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:14.866390 20336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.878286 20336 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:46737
I20260620 15:00:14.878671 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:46737 every 8 connection(s)
I20260620 15:00:14.879317 20336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:14.879559 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 20336
I20260620 15:00:14.880090 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.46.148:41347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:33515
--local_ip_for_outbound_sockets=127.7.46.131
--tserver_master_addrs=127.7.46.190:33343
--webserver_port=46721
--webserver_interface=127.7.46.131
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.46.148:41347/kms with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:00:14 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:14.898867 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:14.881591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:59671 (local address 127.7.46.190:33343)
0620 15:00:14.881717 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:14.881721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:14.882720 (+ 999us) server_negotiation.cc:407] Connection header received
0620 15:00:14.884038 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:14.884042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:14.884104 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:14.884161 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:14.885923 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.886756 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.888687 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.889065 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.889154 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:14.892904 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:14.892908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:14.892911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:14.892940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:14.896253 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:14.896872 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:14.896875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:14.896877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:14.896970 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:14.897571 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:14.897575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:14.897577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:14.897804 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:14.897890 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:14.898572 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 15:00:14.898701 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:00:14.900760 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:14.882023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33343 (local address 127.7.46.130:59671)
0620 15:00:14.882529 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:14.882589 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:14.883756 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:14.884565 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:14.884574 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:14.884926 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:14.885740 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:14.885763 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.886919 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.886922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:14.888543 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:14.888554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.889354 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.889358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:14.889557 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:14.890226 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:14.890246 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:14.892712 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:14.896388 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:14.896395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:14.896410 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:14.896746 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:14.897105 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:14.897108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:14.897110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:14.897284 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:14.897961 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:14.897967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:14.898284 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 15:00:14.899415 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0620 15:00:14.899738 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20260620 15:00:14.902297 20466 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33343
I20260620 15:00:14.902539 20466 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:14.903092 20466 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
W20260620 15:00:14.904695 20466 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:33343 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:00:14.904807 20466 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:14.904956 20466 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:14.905798 20466 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:14.905952 20466 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
W20260620 15:00:14.906708 20466 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:00:15.038970 20468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:15.039314 20468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:15.039386 20468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:15.045365 20468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:15.045446 20468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:15.045542 20468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:00:15.053252 20468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.46.148:41347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:33515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=46721
--tserver_master_addrs=127.7.46.190:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:15.054520 20468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:15.055927 20468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:15.068679 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:15.068795 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:15.068713 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:15.069519 20468 server_base.cc:1034] running on GCE node
I20260620 15:00:15.069926 20468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:15.070534 20468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:15.071717 20468 hybrid_clock.cc:648] HybridClock initialized: now 1781967615071687 us; error 39 us; skew 500 ppm
Jun 20 15:00:15 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:15.075075 20468 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:00:15.076148 20468 webserver.cc:460] Webserver started at http://127.7.46.131:46721/ using document root <none> and password file <none>
I20260620 15:00:15.076846 20468 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:15.076954 20468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:00:15 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:15.097826 20468 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:00:15.101030 20486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.102584 20468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:15.102751 20468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "d65d90af66e14d02af547a72bdd17d3b"
format_stamp: "Formatted at 2026-06-20 15:00:14 on dist-test-slave-30t4"
server_key: "f26ccbef23f5c04ce480b900120f41da"
server_key_iv: "a0ae16ddc32c567d83756884ed761b69"
server_key_version: "kuduclusterkey@0"
I20260620 15:00:15.103489 20468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:15.124125 20468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:15.128747 20468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:15.128960 20468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:15.129561 20468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:15.130388 20468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:15.130460 20468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.130540 20468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:15.130609 20468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.141431 20468 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:33515
I20260620 15:00:15.142727 20468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:15.148342 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 20468
I20260620 15:00:15.152274 20599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:33515 every 8 connection(s)
Jun 20 15:00:15 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:15.159070 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:15.144869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:39983 (local address 127.7.46.190:33343)
0620 15:00:15.145023 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:15.145027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:15.146102 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 15:00:15.147373 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:15.147377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:15.147435 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:15.147541 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:15.149425 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.150284 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.151950 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.152299 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.152417 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:15.154896 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:15.154901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:15.154903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:15.154924 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:15.156869 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:15.157531 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:15.157534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:15.157536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:15.157625 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:15.157963 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:15.157968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:15.157969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:15.158152 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:15.158256 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:15.158769 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 15:00:15.158891 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:00:15.160377 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:15.145371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:33343 (local address 127.7.46.131:39983)
0620 15:00:15.145937 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:15.145996 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:15.147117 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:15.147728 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:15.147738 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:15.148131 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:15.149253 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:15.149277 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.150400 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.150403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:15.151819 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:15.151831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.152602 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.152605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:15.152746 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:15.153374 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:15.153391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:15.154725 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:15.157051 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:15.157058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:15.157074 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:15.157424 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:15.157716 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:15.157720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:15.157722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:15.157865 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:15.158267 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:15.158276 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:15.158533 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 15:00:15.158728 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 15:00:15.159038 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":155,"threads_started":1}
I20260620 15:00:15.161413 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:15.149066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.7.46.190:33343)
0620 15:00:15.149543 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:15.149547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:15.149807 (+ 260us) server_negotiation.cc:407] Connection header received
0620 15:00:15.150071 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:15.150074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:15.150123 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:15.150187 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:15.152955 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.153768 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.155034 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.155420 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.155513 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:15.157404 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:15.157407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:15.157411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:15.157436 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:15.159326 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:15.159816 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:15.159820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:15.159822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:15.159909 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:15.160211 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:15.160214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:15.160216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:15.160429 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:15.160652 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:15.161147 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 15:00:15.161261 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":87,"threads_started":1}
I20260620 15:00:15.161737 20600 heartbeater.cc:347] Connected to a master server at 127.7.46.190:33343
I20260620 15:00:15.161993 20600 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:15.162596 20600 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
W20260620 15:00:15.164258 20600 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:33343 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:00:15.164395 20600 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:15.164575 20600 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:15.165378 20600 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:15.165568 20600 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
W20260620 15:00:15.166332 20600 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 15:00:15 dist-test-slave-30t4 krb5kdc[19138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:15.668689 20332 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:15.668985 20332 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:15.670152 20134 ts_manager.cc:194] Registered new tserver with Master: 3cda9039ef5d45b6ae6545e2f8342e1a (127.7.46.129:41613)
I20260620 15:00:15.671857 20134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:37987
Jun 20 15:00:15 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:15.906952 20466 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:15.907275 20466 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:15.908170 20134 ts_manager.cc:194] Registered new tserver with Master: 6576c4925ce443b2aa91d908ecbcb568 (127.7.46.130:46737)
I20260620 15:00:15.908994 20134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:59671
Jun 20 15:00:16 dist-test-slave-30t4 krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for HTTP/127.7.46.148@KRBTEST.COM
I20260620 15:00:16.166571 20600 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:16.166857 20600 heartbeater.cc:511] Master 127.7.46.190:33343 requested a full tablet report, sending...
I20260620 15:00:16.167649 20134 ts_manager.cc:194] Registered new tserver with Master: d65d90af66e14d02af547a72bdd17d3b (127.7.46.131:33515)
I20260620 15:00:16.168406 20134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:39983
I20260620 15:00:16.170086 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:16.186986 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.171915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.7.46.190:33343)
0620 15:00:16.176140 (+ 4225us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.176145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.176160 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:00:16.176202 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.176205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.176251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.178084 (+ 1833us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:16.178186 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.179019 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.180217 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.180741 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.181259 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.182717 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:16.182721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:16.182725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:16.182752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:16.184585 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:16.185117 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:16.185120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:16.185122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:16.185209 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:16.185554 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:16.185557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:16.185559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:16.185774 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:16.186584 (+ 810us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:16.186665 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 15:00:16.186792 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4147,"thread_start_us":92,"threads_started":1}
I20260620 15:00:16.191287 20134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:16.196273 20134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:16.222214 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.211209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:46737 (local address 127.0.0.1:55086)
0620 15:00:16.213989 (+ 2780us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:16.214002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:16.214082 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:16.215437 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:16.215443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:16.215466 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:16.215604 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.215611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.217077 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.217080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.219388 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.219400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.221628 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.221632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.221779 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.221819 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:00:16.221923 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:00:16.222047 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2728,"thread_start_us":75,"threads_started":1}
I20260620 15:00:16.223023 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.212106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.7.46.130:46737)
0620 15:00:16.213872 (+ 1766us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.213879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.214163 (+ 284us) server_negotiation.cc:407] Connection header received
0620 15:00:16.214239 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.214244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.214447 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.214589 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:16.215738 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.216940 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.219624 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.220444 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.220570 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.222739 (+ 2169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:16.222796 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 15:00:16.222886 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1670,"thread_start_us":86,"threads_started":1}
I20260620 15:00:16.226759 20398 tablet_service.cc:1467] Processing CreateTablet for tablet e53986f0b550483186528361fd00e83e (DEFAULT_TABLE table=test-table [id=242a49680245411b8bd92af38f175a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:16.227964 20398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53986f0b550483186528361fd00e83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:16.229048 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.212447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.7.46.129:41613)
0620 15:00:16.219968 (+ 7521us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.219976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.221494 (+ 1518us) server_negotiation.cc:407] Connection header received
0620 15:00:16.221635 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.221641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.221860 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.222019 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:16.222392 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.223445 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.227465 (+ 4020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.228256 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.228333 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.228458 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:16.228800 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 15:00:16.228897 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7410,"thread_start_us":90,"threads_started":1}
I20260620 15:00:16.230144 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.211458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:41613 (local address 127.0.0.1:50840)
0620 15:00:16.221397 (+ 9939us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:16.221410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:16.221519 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:16.222155 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:16.222158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:16.222172 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:16.222257 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.222265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.223584 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.223588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.224654 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.224663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.228562 (+ 3899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.228566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.228653 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.228674 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:00:16.229942 (+ 1268us) client_negotiation.cc:240] Negotiation successful
0620 15:00:16.230019 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9881,"thread_start_us":80,"threads_started":1}
I20260620 15:00:16.233059 20252 tablet_service.cc:1467] Processing CreateTablet for tablet e53986f0b550483186528361fd00e83e (DEFAULT_TABLE table=test-table [id=242a49680245411b8bd92af38f175a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:16.234109 20252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53986f0b550483186528361fd00e83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:16.243247 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.210505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:33515 (local address 127.0.0.1:50694)
0620 15:00:16.213423 (+ 2918us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:16.213473 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:16.213621 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:16.236569 (+ 22948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:16.236572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:16.236592 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:16.236686 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.236691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.238295 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.238299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.239456 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.239468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.242816 (+ 3348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.242820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.242926 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.242956 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:00:16.243017 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 15:00:16.243092 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2748,"thread_start_us":88,"threads_started":1}
I20260620 15:00:16.243996 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.210601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.7.46.131:33515)
0620 15:00:16.236068 (+ 25467us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.236075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.236103 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:00:16.236182 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.236187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.236383 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.236840 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:16.236992 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.238156 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.239618 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.240427 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.243538 (+ 3111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.243695 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:16.243747 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 15:00:16.243857 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25350,"thread_start_us":80,"threads_started":1}
I20260620 15:00:16.246160 20637 tablet_bootstrap.cc:492] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a: Bootstrap starting.
I20260620 15:00:16.246850 20534 tablet_service.cc:1467] Processing CreateTablet for tablet e53986f0b550483186528361fd00e83e (DEFAULT_TABLE table=test-table [id=242a49680245411b8bd92af38f175a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:16.247915 20534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53986f0b550483186528361fd00e83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:16.248788 20637 tablet_bootstrap.cc:654] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:16.249742 20637 log.cc:824] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:16.252537 20637 tablet_bootstrap.cc:492] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a: No bootstrap required, opened a new log
I20260620 15:00:16.252727 20637 ts_tablet_manager.cc:1397] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 15:00:16.257150 20637 raft_consensus.cc:348] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.257524 20637 raft_consensus.cc:374] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:16.257611 20637 raft_consensus.cc:729] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda9039ef5d45b6ae6545e2f8342e1a, State: Initialized, Role: FOLLOWER
I20260620 15:00:16.257656 20636 tablet_bootstrap.cc:492] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: Bootstrap starting.
I20260620 15:00:16.258143 20637 consensus_queue.cc:260] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.259238 20637 ts_tablet_manager.cc:1428] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260620 15:00:16.259634 20332 heartbeater.cc:503] Master 127.7.46.190:33343 was elected leader, sending a full tablet report...
I20260620 15:00:16.260138 20636 tablet_bootstrap.cc:654] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:16.266115 20639 tablet_bootstrap.cc:492] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b: Bootstrap starting.
I20260620 15:00:16.267165 20636 log.cc:824] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:16.269594 20639 tablet_bootstrap.cc:654] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:16.270828 20639 log.cc:824] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:16.273185 20640 raft_consensus.cc:482] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:16.273365 20640 raft_consensus.cc:504] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.274580 20640 leader_election.cc:288] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d65d90af66e14d02af547a72bdd17d3b (127.7.46.131:33515), 6576c4925ce443b2aa91d908ecbcb568 (127.7.46.130:46737)
I20260620 15:00:16.277334 20636 tablet_bootstrap.cc:492] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: No bootstrap required, opened a new log
I20260620 15:00:16.277699 20636 ts_tablet_manager.cc:1397] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260620 15:00:16.281019 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.275124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:40501 (local address 127.7.46.131:33515)
0620 15:00:16.275260 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.275265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.276642 (+ 1377us) server_negotiation.cc:407] Connection header received
0620 15:00:16.276776 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.276779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.276830 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.276911 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:16.277347 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.278124 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.279678 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.280288 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.280385 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.280442 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:16.280807 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 15:00:16.280884 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:00:16.281606 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.275033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:33515 (local address 127.7.46.129:40501)
0620 15:00:16.276553 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:16.276568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:16.276682 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:16.277041 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:16.277045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:16.277064 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:16.277204 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.277211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.278275 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.278278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.279530 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.279541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.280545 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.280549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.280660 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.280685 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:00:16.281361 (+ 676us) client_negotiation.cc:240] Negotiation successful
0620 15:00:16.281441 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"thread_start_us":113,"threads_started":1}
I20260620 15:00:16.282106 20636 raft_consensus.cc:348] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.285960 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.275748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:51343 (local address 127.7.46.130:46737)
0620 15:00:16.275882 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.275886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.277030 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 15:00:16.280688 (+ 3658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.280692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.280743 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.280792 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:16.281183 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.281884 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.284634 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.285260 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.285354 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.285400 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:16.285747 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 15:00:16.285820 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:00:16.290735 20636 raft_consensus.cc:374] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:16.291298 20636 raft_consensus.cc:729] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6576c4925ce443b2aa91d908ecbcb568, State: Initialized, Role: FOLLOWER
I20260620 15:00:16.286240 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.275428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:46737 (local address 127.7.46.129:51343)
0620 15:00:16.276925 (+ 1497us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:16.276954 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:16.280350 (+ 3396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:16.280918 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:16.280921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:16.280951 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:16.281060 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.281066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.282238 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.282241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.283425 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.283436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.285511 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.285515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.285620 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.285639 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 15:00:16.286036 (+ 397us) client_negotiation.cc:240] Negotiation successful
0620 15:00:16.286109 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1425,"thread_start_us":82,"threads_started":1}
I20260620 15:00:16.292619 20420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53986f0b550483186528361fd00e83e" candidate_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6576c4925ce443b2aa91d908ecbcb568" is_pre_election: true
I20260620 15:00:16.293975 20639 tablet_bootstrap.cc:492] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b: No bootstrap required, opened a new log
I20260620 15:00:16.294188 20639 ts_tablet_manager.cc:1397] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20260620 15:00:16.293955 20636 consensus_queue.cc:260] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.294512 20420 raft_consensus.cc:2454] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cda9039ef5d45b6ae6545e2f8342e1a in term 0.
I20260620 15:00:16.295467 20222 leader_election.cc:302] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cda9039ef5d45b6ae6545e2f8342e1a, 6576c4925ce443b2aa91d908ecbcb568; no voters:
I20260620 15:00:16.295779 20640 raft_consensus.cc:2781] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:16.295852 20640 raft_consensus.cc:482] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:16.295897 20640 raft_consensus.cc:3037] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:16.297174 20640 raft_consensus.cc:504] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.297541 20640 leader_election.cc:288] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [CANDIDATE]: Term 1 election: Requested vote from peers d65d90af66e14d02af547a72bdd17d3b (127.7.46.131:33515), 6576c4925ce443b2aa91d908ecbcb568 (127.7.46.130:46737)
I20260620 15:00:16.298144 20420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53986f0b550483186528361fd00e83e" candidate_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6576c4925ce443b2aa91d908ecbcb568"
I20260620 15:00:16.298171 20553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53986f0b550483186528361fd00e83e" candidate_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d65d90af66e14d02af547a72bdd17d3b"
I20260620 15:00:16.298251 20420 raft_consensus.cc:3037] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:16.298797 20554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53986f0b550483186528361fd00e83e" candidate_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d65d90af66e14d02af547a72bdd17d3b" is_pre_election: true
W20260620 15:00:16.299427 20222 leader_election.cc:341] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d65d90af66e14d02af547a72bdd17d3b (127.7.46.131:33515): Illegal state: must be running to vote when last-logged opid is not known
I20260620 15:00:16.299512 20420 raft_consensus.cc:2454] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cda9039ef5d45b6ae6545e2f8342e1a in term 1.
W20260620 15:00:16.299638 20222 leader_election.cc:341] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d65d90af66e14d02af547a72bdd17d3b (127.7.46.131:33515): Illegal state: must be running to vote when last-logged opid is not known
I20260620 15:00:16.300002 20222 leader_election.cc:302] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cda9039ef5d45b6ae6545e2f8342e1a, 6576c4925ce443b2aa91d908ecbcb568; no voters: d65d90af66e14d02af547a72bdd17d3b
I20260620 15:00:16.300164 20640 raft_consensus.cc:2781] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:16.300381 20640 raft_consensus.cc:686] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [term 1 LEADER]: Becoming Leader. State: Replica: 3cda9039ef5d45b6ae6545e2f8342e1a, State: Running, Role: LEADER
I20260620 15:00:16.300684 20640 consensus_queue.cc:237] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.300818 20639 raft_consensus.cc:348] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.301023 20639 raft_consensus.cc:374] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:16.301070 20639 raft_consensus.cc:729] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d65d90af66e14d02af547a72bdd17d3b, State: Initialized, Role: FOLLOWER
I20260620 15:00:16.301527 20639 consensus_queue.cc:260] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } }
I20260620 15:00:16.302568 20639 ts_tablet_manager.cc:1428] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:00:16.302889 20600 heartbeater.cc:503] Master 127.7.46.190:33343 was elected leader, sending a full tablet report...
I20260620 15:00:16.304194 20134 catalog_manager.cc:5495] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cda9039ef5d45b6ae6545e2f8342e1a (127.7.46.129). New cstate: current_term: 1 leader_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cda9039ef5d45b6ae6545e2f8342e1a" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 41613 } health_report { overall_health: HEALTHY } } }
I20260620 15:00:16.310621 20636 ts_tablet_manager.cc:1428] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260620 15:00:16.310973 20466 heartbeater.cc:503] Master 127.7.46.190:33343 was elected leader, sending a full tablet report...
I20260620 15:00:16.330818 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.325964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.7.46.129:41613)
0620 15:00:16.326101 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.326105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.326121 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:00:16.326203 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.326206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.326268 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.326376 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:16.326739 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.327521 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.328773 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.329100 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.329190 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.329581 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:16.330095 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:16.330204 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:16.330597 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 15:00:16.330655 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:00:16.337942 20554 raft_consensus.cc:3037] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:16.338032 20420 raft_consensus.cc:1261] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 1 FOLLOWER]: Refusing update from remote peer 3cda9039ef5d45b6ae6545e2f8342e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:16.338781 20640 consensus_queue.cc:1035] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6576c4925ce443b2aa91d908ecbcb568" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 46737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:16.339396 20554 raft_consensus.cc:1261] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b [term 1 FOLLOWER]: Refusing update from remote peer 3cda9039ef5d45b6ae6545e2f8342e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:16.340202 20640 consensus_queue.cc:1035] T e53986f0b550483186528361fd00e83e P 3cda9039ef5d45b6ae6545e2f8342e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d65d90af66e14d02af547a72bdd17d3b" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:16.354735 20650 mvcc.cc:205] Tried to move back new op lower bound from 7298939356511940608 to 7298939356371267584. Current Snapshot: MvccSnapshot[applied={T|T < 7298939356511940608}]
I20260620 15:00:16.355435 20653 mvcc.cc:205] Tried to move back new op lower bound from 7298939356511940608 to 7298939356371267584. Current Snapshot: MvccSnapshot[applied={T|T < 7298939356511940608}]
I20260620 15:00:16.363350 20651 mvcc.cc:205] Tried to move back new op lower bound from 7298939356511940608 to 7298939356371267584. Current Snapshot: MvccSnapshot[applied={T|T < 7298939356511940608}]
I20260620 15:00:16.364745 20134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:16.364913 20134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:16.367456 20134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41282de409a44460aaf6a0dbae190be3: Sending DeleteTablet for 3 replicas of tablet e53986f0b550483186528361fd00e83e
I20260620 15:00:16.368309 20252 tablet_service.cc:1514] Processing DeleteTablet for tablet e53986f0b550483186528361fd00e83e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:16 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:50840
I20260620 15:00:16.368513 20534 tablet_service.cc:1514] Processing DeleteTablet for tablet e53986f0b550483186528361fd00e83e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:16 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:50694
I20260620 15:00:16.368970 20398 tablet_service.cc:1514] Processing DeleteTablet for tablet e53986f0b550483186528361fd00e83e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:16 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:55086
I20260620 15:00:16.369403 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 20172
I20260620 15:00:16.370589 20662 tablet_replica.cc:330] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: stopping tablet replica
I20260620 15:00:16.370936 20662 raft_consensus.cc:2229] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:16.371330 20662 raft_consensus.cc:2258] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:16.372399 20661 tablet_replica.cc:330] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b: stopping tablet replica
I20260620 15:00:16.372706 20661 raft_consensus.cc:2229] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:16.373003 20661 raft_consensus.cc:2258] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:16.377110 20662 ts_tablet_manager.cc:1905] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:16.379679 20662 ts_tablet_manager.cc:1918] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:16.379837 20662 log.cc:1196] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/e53986f0b550483186528361fd00e83e
I20260620 15:00:16.380187 20662 ts_tablet_manager.cc:1939] T e53986f0b550483186528361fd00e83e P 6576c4925ce443b2aa91d908ecbcb568: Deleting consensus metadata
I20260620 15:00:16.381372 20121 catalog_manager.cc:4841] TS 6576c4925ce443b2aa91d908ecbcb568 (127.7.46.130:46737): tablet e53986f0b550483186528361fd00e83e (table test-table [id=242a49680245411b8bd92af38f175a52]) successfully deleted
W20260620 15:00:16.383338 20121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:41613 (error 108)
I20260620 15:00:16.384166 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 20336
I20260620 15:00:16.384394 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.384064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:41613 (local address 127.0.0.1:50862)
0620 15:00:16.384244 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:16.384315 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 15:00:16.384684 20121 catalog_manager.cc:4584] TS 3cda9039ef5d45b6ae6545e2f8342e1a (127.7.46.129:41613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e53986f0b550483186528361fd00e83e: Network error: Client connection negotiation failed: client connection to 127.7.46.129:41613: connect: Connection refused (error 111)
I20260620 15:00:16.393057 20661 ts_tablet_manager.cc:1905] T e53986f0b550483186528361fd00e83e P d65d90af66e14d02af547a72bdd17d3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:16.394130 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 20468
I20260620 15:00:16.402420 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 20101
I20260620 15:00:16.410441 7354 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:00:16.744742 7354 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:00:16.744849 7354 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:00:17.089599 7354 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:00:17.089 UTC [19168] LOG: received smart shutdown request
2026-06-20 15:00:17.093 UTC [19168] LOG: background worker "logical replication launcher" (PID 19177) exited with exit code 1
2026-06-20 15:00:17.093 UTC [19172] LOG: shutting down
2026-06-20 15:00:17.246 UTC [19168] LOG: database system is shut down
2026-06-20T15:00:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47344 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:17 dist-test-slave-30t4 krb5kdc[20672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:17 dist-test-slave-30t4 krb5kdc[20672](info): set up 2 sockets
Jun 20 15:00:17 dist-test-slave-30t4 krb5kdc[20672](info): commencing operation
krb5kdc: starting...
W20260620 15:00:18.411834 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:00:18 dist-test-slave-30t4 krb5kdc[20672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:00:18.532970 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43895
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43895,127.7.46.189:36161,127.7.46.188:34551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:18.641258 20685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:18.641564 20685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:18.641654 20685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:18.645715 20685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:18.645802 20685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:18.645840 20685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:18.645901 20685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:18.645941 20685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:18.651129 20685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43895,127.7.46.189:36161,127.7.46.188:34551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:18.652277 20685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:18.653368 20685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:18.661790 20691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:18.661837 20685 server_base.cc:1034] running on GCE node
W20260620 15:00:18.661792 20690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:18.661876 20693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:18.662619 20685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:18.663645 20685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:18.664804 20685 hybrid_clock.cc:648] HybridClock initialized: now 1781967618664785 us; error 29 us; skew 500 ppm
Jun 20 15:00:18 dist-test-slave-30t4 krb5kdc[20672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:18.667665 20685 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:00:18.668553 20685 webserver.cc:460] Webserver started at http://127.7.46.190:40739/ using document root <none> and password file <none>
I20260620 15:00:18.669049 20685 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:18.669106 20685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:18.669263 20685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:18.671244 20685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "d33af4b7b3774f648f2d179a3136554a"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-30t4"
server_key: "9b77c8d351ebff2f2259c897bf6f08b3"
server_key_iv: "618585da10e9b1c5faa371a91f9ce771"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.671751 20685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "d33af4b7b3774f648f2d179a3136554a"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-30t4"
server_key: "9b77c8d351ebff2f2259c897bf6f08b3"
server_key_iv: "618585da10e9b1c5faa371a91f9ce771"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.675367 20685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:00:18.677721 20700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:18.678994 20685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:18.679134 20685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "d33af4b7b3774f648f2d179a3136554a"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-30t4"
server_key: "9b77c8d351ebff2f2259c897bf6f08b3"
server_key_iv: "618585da10e9b1c5faa371a91f9ce771"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.679242 20685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:18.690379 20685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:18.693707 20685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:18.693915 20685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:18.701141 20685 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:43895
I20260620 15:00:18.701164 20752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:43895 every 8 connection(s)
I20260620 15:00:18.702067 20685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:18.704739 20753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:18.709085 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 20685
I20260620 15:00:18.709249 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:00:18.709585 7354 external_mini_cluster.cc:1428] Setting key b15de2f97bc1d5050873e2bd95452299
I20260620 15:00:18.709408 20753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.46.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.189@KRBTEST.COM" created.
Jun 20 15:00:18 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:18.725487 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.711518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43895 (local address 127.0.0.1:48702)
0620 15:00:18.711995 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.712046 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:18.712901 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:18.713733 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:18.713738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:18.714185 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:18.714412 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:18.714424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.715548 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.715553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:18.716320 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:18.716329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.716989 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.716995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:18.717144 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.717793 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:18.717815 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:18.720235 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:18.722384 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:18.722394 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:18.722403 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:18.722606 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:18.723049 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:18.723053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:18.723055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:18.723179 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:18.723813 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:18.723819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:18.723980 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 15:00:18.724144 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 15:00:18.724520 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":110,"thread_start_us":126,"threads_started":1}
I20260620 15:00:18.725492 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.711519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.7.46.190:43895)
0620 15:00:18.713229 (+ 1710us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:18.713237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:18.713256 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:00:18.713368 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:18.713372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:18.713555 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:18.713736 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:18.714559 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.715405 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.716434 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.716797 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.716955 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.720399 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:18.720405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:18.720414 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:18.720433 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:18.722281 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.722718 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.722726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.722732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.722868 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.723290 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.723293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.723295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.723682 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:18.723846 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:18.724213 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 15:00:18.724457 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":181,"threads_started":1}
I20260620 15:00:18.727874 20753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:00:18.728583 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.728343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35880)
0620 15:00:18.728450 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.728515 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:00:18.728807 20701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
I20260620 15:00:18.729588 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.729373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35884)
0620 15:00:18.729478 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.729535 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:00:18.729717 20753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:36161: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab.
Entry for principal kudu/127.7.46.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab.
I20260620 15:00:18.735152 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.189:36161
--webserver_interface=127.7.46.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab
--principal=kudu/127.7.46.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43895,127.7.46.189:36161,127.7.46.188:34551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:18.778962 20753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } attempt: 1
I20260620 15:00:18.780045 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.779663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35900)
0620 15:00:18.779903 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.779966 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 15:00:18.780797 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.780591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35914)
0620 15:00:18.780700 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.780745 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:00:18.780975 20753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:36161: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
W20260620 15:00:18.848726 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:18.849005 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:18.849050 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:00:18.849175 20753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } attempt: 2
I20260620 15:00:18.850019 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.849731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35918)
0620 15:00:18.849883 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.849944 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:00:18.850744 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.850553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35924)
0620 15:00:18.850652 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.850695 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 15:00:18.850973 20753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:36161: Network error: Client connection negotiation failed: client connection to 127.7.46.189:36161: connect: Connection refused (error 111)
W20260620 15:00:18.853822 20760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:18.853878 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:18.853900 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:18.853920 20760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:18.853982 20760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:18.859156 20760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43895,127.7.46.189:36161,127.7.46.188:34551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab
--principal=kudu/127.7.46.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.189:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.46.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:18.860287 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:18.861274 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:18.869668 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:18.869719 20760 server_base.cc:1034] running on GCE node
W20260620 15:00:18.869714 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:18.869684 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:18.870440 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:18.871521 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:18.872680 20760 hybrid_clock.cc:648] HybridClock initialized: now 1781967618872659 us; error 34 us; skew 500 ppm
Jun 20 15:00:18 dist-test-slave-30t4 krb5kdc[20672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:18.875275 20760 init.cc:377] Logged in from keytab as kudu/127.7.46.189@KRBTEST.COM (short username kudu)
I20260620 15:00:18.876128 20760 webserver.cc:460] Webserver started at http://127.7.46.189:38009/ using document root <none> and password file <none>
I20260620 15:00:18.876667 20760 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:18.876729 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:18.876891 20760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:18.878873 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/data/instance:
uuid: "c64a31b7fe6c4a3e9b56dad711d82b52"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-30t4"
server_key: "1823318ee6f1511b11afa2ac9116b2c7"
server_key_iv: "d35c97ab671596b61d2e178eef84a313"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.879374 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/wal/instance:
uuid: "c64a31b7fe6c4a3e9b56dad711d82b52"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-30t4"
server_key: "1823318ee6f1511b11afa2ac9116b2c7"
server_key_iv: "d35c97ab671596b61d2e178eef84a313"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.882912 20760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 15:00:18.885337 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:18.886513 20760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:00:18.886633 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/wal
uuid: "c64a31b7fe6c4a3e9b56dad711d82b52"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-30t4"
server_key: "1823318ee6f1511b11afa2ac9116b2c7"
server_key_iv: "d35c97ab671596b61d2e178eef84a313"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.886729 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:18.905285 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:18.908165 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:18.908373 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:18.915380 20760 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.189:36161
I20260620 15:00:18.915412 20827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.189:36161 every 8 connection(s)
I20260620 15:00:18.916380 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
I20260620 15:00:18.919039 20828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:18.921197 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 20760
I20260620 15:00:18.921361 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-1/wal/instance
I20260620 15:00:18.921660 7354 external_mini_cluster.cc:1428] Setting key 32091ba4ccdb7b313b858886bb3c98ed
I20260620 15:00:18.922945 20828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.46.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.188@KRBTEST.COM" created.
Jun 20 15:00:18 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:18.937801 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.925330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.7.46.190:43895)
0620 15:00:18.925523 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:18.925531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:18.926371 (+ 840us) server_negotiation.cc:407] Connection header received
0620 15:00:18.927297 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:18.927302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:18.927367 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:18.927491 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:18.928807 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.929687 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.930972 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.931227 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.931318 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.933898 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:18.933901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:18.933903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:18.933927 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:18.935459 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.936118 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.936121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.936122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.936197 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.936634 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.936636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.936637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.936809 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:18.936882 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:18.937577 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 15:00:18.937677 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:00:18.939131 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.925748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43895 (local address 127.0.0.1:48708)
0620 15:00:18.926200 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.926251 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:18.927099 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:18.927650 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:18.927657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:18.928163 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:18.928643 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:18.928656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.929850 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.929853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:18.930851 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:18.930861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.931363 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.931366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:18.931502 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.932197 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:18.932213 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:18.933710 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:18.935583 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:18.935590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:18.935608 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:18.936012 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:18.936293 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:18.936299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:18.936303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:18.936494 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:18.936917 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:18.936927 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:18.937299 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 15:00:18.937513 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 15:00:18.937873 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":123,"threads_started":1}
I20260620 15:00:18.940658 20828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:00:18.942147 20753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } attempt: 3
Jun 20 15:00:18 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
Jun 20 15:00:18 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab.
Entry for principal kudu/127.7.46.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab.
I20260620 15:00:18.950613 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.188:34551
--webserver_interface=127.7.46.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab
--principal=kudu/127.7.46.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43895,127.7.46.189:36161,127.7.46.188:34551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:18.951941 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.941162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35936)
0620 15:00:18.941289 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.941301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:18.941440 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:18.942120 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:18.942123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:18.942300 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:18.942416 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:18.942427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.943508 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.943514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:18.944401 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:18.944410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.945135 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.945139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:18.945264 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.945714 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:18.945727 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:18.947210 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:18.950243 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:18.950246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:18.950247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:18.950426 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:18.950774 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:18.950779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:18.950782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:18.950880 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:18.951450 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:18.951452 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:18.951534 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 15:00:18.951616 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 15:00:18.951713 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":3}
I20260620 15:00:18.952608 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.941303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.7.46.189:36161)
0620 15:00:18.941631 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:18.941640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:18.941658 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:00:18.941693 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:18.941696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:18.941839 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:18.941949 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:18.942567 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.943392 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.944502 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.945018 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.945131 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.947365 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:18.947374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:18.947392 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:18.947436 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:18.950129 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.950537 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.950544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.950546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.950662 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.950981 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.950987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.950992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.951341 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:18.951481 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:18.951749 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 15:00:18.952423 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":233,"thread_start_us":162,"threads_started":1}
I20260620 15:00:18.954507 20828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:00:18.955264 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.955019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38410)
0620 15:00:18.955132 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.955212 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:00:18.955442 20778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
I20260620 15:00:18.956101 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.942537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35944)
0620 15:00:18.942664 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.942677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:18.942742 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:18.943353 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:18.943356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:18.943593 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:18.944230 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:18.944240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.945229 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.945233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:18.946138 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:18.946152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.946875 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.946878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:18.946992 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.947500 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:18.947515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:18.949426 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:18.954125 (+ 4699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:18.954129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:18.954131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:18.954384 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:18.954753 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:18.954759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:18.954765 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:18.954869 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:18.955398 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:18.955403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:18.955524 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 15:00:18.955645 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 15:00:18.955844 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:00:18.956516 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.956140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38422)
0620 15:00:18.956426 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.956468 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
W20260620 15:00:18.956672 20828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:34551: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
I20260620 15:00:18.956866 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.942869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.7.46.189:36161)
0620 15:00:18.943169 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:18.943176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:18.943186 (+ 10us) server_negotiation.cc:407] Connection header received
0620 15:00:18.943211 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:18.943213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:18.943241 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:18.943337 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:18.944359 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.945133 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.946312 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.946702 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.946811 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.949600 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:18.949606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:18.949609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:18.949633 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:18.953242 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.954480 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.954486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.954491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.954636 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.954983 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.954990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.955003 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.955241 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:18.955346 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:18.956396 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0620 15:00:18.956529 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":148,"threads_started":1}
I20260620 15:00:18.957504 20753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:00:18.958194 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.957970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38438)
0620 15:00:18.958096 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.958139 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 15:00:18.958849 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.958626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38446)
0620 15:00:18.958763 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.958804 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 15:00:18.958993 20753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:34551: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
I20260620 15:00:18.990163 20753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } attempt: 1
I20260620 15:00:18.991019 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.990734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38452)
0620 15:00:18.990875 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.990945 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:00:18.991714 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.991510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38456)
0620 15:00:18.991614 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:18.991664 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 15:00:18.991855 20753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:34551: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
I20260620 15:00:19.005947 20828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } attempt: 1
I20260620 15:00:19.006726 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.006437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38462)
0620 15:00:19.006572 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.006646 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:00:19.007521 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.007289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38466)
0620 15:00:19.007413 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.007467 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:00:19.007640 20828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:34551: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
W20260620 15:00:19.062831 20835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:19.063131 20835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:19.063228 20835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:00:19.067018 20753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } attempt: 2
W20260620 15:00:19.067539 20835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:19.067618 20835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:19.067662 20835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:19.067709 20835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:19.067754 20835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:19.067886 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.067509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38472)
0620 15:00:19.067663 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.067756 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 15:00:19.069311 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.068902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38478)
0620 15:00:19.069087 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.069179 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 15:00:19.069484 20753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:34551: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
I20260620 15:00:19.073582 20835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43895,127.7.46.189:36161,127.7.46.188:34551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab
--principal=kudu/127.7.46.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.188:34551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.46.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:19.074601 20835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:19.075666 20835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:00:19.075804 20828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } attempt: 2
I20260620 15:00:19.076480 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.076222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38494)
0620 15:00:19.076345 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.076402 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:00:19.077260 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.077067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38496)
0620 15:00:19.077169 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.077212 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:00:19.077404 20828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:34551: Network error: Client connection negotiation failed: client connection to 127.7.46.188:34551: connect: Connection refused (error 111)
W20260620 15:00:19.084296 20844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:19.084261 20841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:19.084272 20842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:19.084628 20835 server_base.cc:1034] running on GCE node
I20260620 15:00:19.085181 20835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:19.086283 20835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:19.087464 20835 hybrid_clock.cc:648] HybridClock initialized: now 1781967619087438 us; error 53 us; skew 500 ppm
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:19.090332 20835 init.cc:377] Logged in from keytab as kudu/127.7.46.188@KRBTEST.COM (short username kudu)
I20260620 15:00:19.091241 20835 webserver.cc:460] Webserver started at http://127.7.46.188:34457/ using document root <none> and password file <none>
I20260620 15:00:19.091850 20835 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:19.091915 20835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:19.092145 20835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:19.094141 20835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/data/instance:
uuid: "66c42bbfe0044a48ae6c05bfa4f40e23"
format_stamp: "Formatted at 2026-06-20 15:00:19 on dist-test-slave-30t4"
server_key: "46f26c8da96f4e6f20990796e3a1e2b9"
server_key_iv: "a452867b65e577357544753510325e82"
server_key_version: "encryptionkey@0"
I20260620 15:00:19.094707 20835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/wal/instance:
uuid: "66c42bbfe0044a48ae6c05bfa4f40e23"
format_stamp: "Formatted at 2026-06-20 15:00:19 on dist-test-slave-30t4"
server_key: "46f26c8da96f4e6f20990796e3a1e2b9"
server_key_iv: "a452867b65e577357544753510325e82"
server_key_version: "encryptionkey@0"
I20260620 15:00:19.098536 20835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:00:19.101011 20851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:19.102139 20835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:00:19.102265 20835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/wal
uuid: "66c42bbfe0044a48ae6c05bfa4f40e23"
format_stamp: "Formatted at 2026-06-20 15:00:19 on dist-test-slave-30t4"
server_key: "46f26c8da96f4e6f20990796e3a1e2b9"
server_key_iv: "a452867b65e577357544753510325e82"
server_key_version: "encryptionkey@0"
I20260620 15:00:19.102371 20835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:19.121393 20835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:19.124691 20835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:19.124899 20835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:19.131914 20835 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.188:34551
I20260620 15:00:19.131943 20903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.188:34551 every 8 connection(s)
I20260620 15:00:19.132825 20835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
I20260620 15:00:19.135377 20904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:19.137753 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 20835
I20260620 15:00:19.137884 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967464421185-7354-0/minicluster-data/master-2/wal/instance
I20260620 15:00:19.138171 7354 external_mini_cluster.cc:1428] Setting key 6cd846a7834564450ab32dbcc98bc893
I20260620 15:00:19.139285 20904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
I20260620 15:00:19.151150 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.141144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.7.46.190:43895)
0620 15:00:19.141295 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.141300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.141611 (+ 311us) server_negotiation.cc:407] Connection header received
0620 15:00:19.141790 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.141793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.141838 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.141888 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.142796 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.143277 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.144346 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.144562 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.144638 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.147198 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.147201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.147204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.147237 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.148937 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.149578 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.149581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.149583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.149662 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.150030 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.150036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.150040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.150256 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.150352 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.150845 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.150978 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:00:19.152060 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.141087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43895 (local address 127.0.0.1:48718)
0620 15:00:19.141503 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.141548 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:19.141686 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:19.142057 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:19.142061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:19.142468 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:19.142654 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.142665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.143382 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.143387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.144202 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.144211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.144693 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.144697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.144903 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.145666 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:19.145681 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:19.147034 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:19.149061 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.149069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.149076 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.149458 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.149755 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.149758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.149760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.149940 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.150364 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:19.150369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:19.150589 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 15:00:19.150772 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 15:00:19.151203 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":55,"spinlock_wait_cycles":6272,"thread_start_us":152,"threads_started":1}
I20260620 15:00:19.152071 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.139474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.7.46.188:34551)
0620 15:00:19.139936 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.139955 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.140004 (+ 49us) server_negotiation.cc:407] Connection header received
0620 15:00:19.140806 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.140823 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.141183 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.141541 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.142288 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.143418 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.144286 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.144699 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.144906 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.147677 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.147683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.147691 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.147730 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.149449 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.149912 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.149916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.149917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.150013 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.150280 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.150286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.150291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.150580 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.150715 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.150998 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.151413 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":302,"server-negotiator.queue_time_us":329,"thread_start_us":141,"threads_started":1}
I20260620 15:00:19.153244 20904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:19.161712 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.153744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:36161 (local address 127.0.0.1:35948)
0620 15:00:19.153855 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.153869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:19.153947 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:19.154213 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:19.154216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:19.154393 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:19.154512 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.154519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.155404 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.155407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.156055 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.156065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.156508 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.156512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.156604 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.157067 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:19.157078 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:19.158281 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:19.160248 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.160251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.160253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.160527 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.160783 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.160788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.160793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.160898 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.161242 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:19.161245 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:19.161341 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 15:00:19.161446 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:00:19.161563 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 15:00:19.161746 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.153870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.7.46.189:36161)
0620 15:00:19.154000 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.154006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.154018 (+ 12us) server_negotiation.cc:407] Connection header received
0620 15:00:19.154064 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.154066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.154106 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.154167 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.154644 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.155289 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.156171 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.156404 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.156482 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.158410 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.158414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.158416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.158443 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.160135 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.160622 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.160625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.160626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.160693 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.160989 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.160992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.160994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.161144 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.161251 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.161506 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.161612 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:00:19.162433 20904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:00:19.162914 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.154494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.7.46.190:43895)
0620 15:00:19.154619 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.154623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.154634 (+ 11us) server_negotiation.cc:407] Connection header received
0620 15:00:19.154696 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.154702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.154741 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.154821 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.155325 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.155867 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.156782 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.157106 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.157184 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.159477 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.159480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.159483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.159506 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.161103 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.161497 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.161502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.161503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.161583 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.162044 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.162047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.162049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.162245 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.162484 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.162616 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.162740 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
I20260620 15:00:19.168566 20828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } attempt: 3
I20260620 15:00:19.168676 20753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } attempt: 3
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
I20260620 15:00:19.172438 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.162896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38522)
0620 15:00:19.163049 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.163066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:19.163128 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:19.163353 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:19.163354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:19.163505 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:19.163590 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.163595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.165037 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.165041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.165827 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.165838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.166386 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.166389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.166461 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.166891 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:19.166905 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:19.167981 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:19.169903 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.169906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.169908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.170133 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.170398 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.170400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.170402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.170507 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.171992 (+ 1485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:19.171995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:19.172063 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 15:00:19.172144 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 15:00:19.172247 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":2}
I20260620 15:00:19.172863 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.163094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.7.46.188:34551)
0620 15:00:19.163183 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.163189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.163201 (+ 12us) server_negotiation.cc:407] Connection header received
0620 15:00:19.163232 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.163235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.163270 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.163347 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.163710 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.164937 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.165941 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.166286 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.166383 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.168119 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.168125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.168131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.168163 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.169777 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.170227 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.170230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.170232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.170305 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.170864 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.170870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.170872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.171879 (+ 1007us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.171963 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.172603 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.172695 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":33}
I20260620 15:00:19.173607 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.163981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.7.46.189:36161)
0620 15:00:19.164082 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.164086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.164154 (+ 68us) server_negotiation.cc:407] Connection header received
0620 15:00:19.164276 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.164278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.164306 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.164363 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.165037 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.165748 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.166710 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.166935 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.167040 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.170091 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.170096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.170099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.170123 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.171670 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.172144 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.172148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.172150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.172226 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.172622 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.172628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.172630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.172817 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.173008 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.173347 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.173457 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
Jun 20 15:00:19 dist-test-slave-30t4 krb5kdc[20672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
I20260620 15:00:19.176609 20904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: Bootstrap starting.
I20260620 15:00:19.178016 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.169027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38524)
0620 15:00:19.169213 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.169226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:19.169296 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:19.169779 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:19.169783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:19.169966 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:19.170061 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.170069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.171436 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.171440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.172267 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.172277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.172849 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.172852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.172950 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.173363 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:19.173376 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:19.174686 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:19.176508 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.176512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.176514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.176747 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.177014 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.177017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.177019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.177102 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.177533 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:19.177538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:19.177630 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 15:00:19.177737 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:00:19.177863 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 15:00:19.178304 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.169235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.7.46.188:34551)
0620 15:00:19.169551 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.169560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.169575 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:00:19.169610 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.169612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.169647 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.170170 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.170261 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.171314 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.172387 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.172724 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.174788 (+ 2064us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.174865 (+ 77us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.174867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.174869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.174901 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.176398 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.176828 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.176832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.176834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.176915 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.177197 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.177200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.177203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.177427 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.177531 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.177786 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.177898 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":100,"threads_started":1}
I20260620 15:00:19.179172 20904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:19.180090 20904 log.cc:824] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:19.181080 20753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: Bootstrap starting.
I20260620 15:00:19.182546 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:00:19.182687 20904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: No bootstrap required, opened a new log
I20260620 15:00:19.183602 20753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:19.184262 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.169425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:34551 (local address 127.0.0.1:38530)
0620 15:00:19.169548 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:19.169563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:19.169629 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:19.170267 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:19.170270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:19.170452 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:19.170571 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.170581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.171600 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.171604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.172397 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:19.172405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.173755 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.173760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:19.173864 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.174302 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:19.174316 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:19.175614 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:19.182483 (+ 6869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.182486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.182488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.182722 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.183165 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:19.183169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:19.183171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:19.183259 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:19.183670 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:19.183673 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:19.183847 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 15:00:19.183954 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:00:19.184097 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:00:19.184468 20753 log.cc:824] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:19.185058 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.169759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.7.46.188:34551)
0620 15:00:19.170040 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.170049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.170061 (+ 12us) server_negotiation.cc:407] Connection header received
0620 15:00:19.170091 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.170095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.170140 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.170676 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.170789 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.171504 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.172753 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.173252 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.180532 (+ 7280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.180629 (+ 97us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.180632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.180635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.180655 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.182365 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.182891 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.182894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.182896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.182980 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.183354 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.183357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.183359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.183538 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.183626 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.184781 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.184890 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260620 15:00:19.186493 20753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: No bootstrap required, opened a new log
I20260620 15:00:19.187521 20904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.187742 20904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:19.187863 20904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c42bbfe0044a48ae6c05bfa4f40e23, State: Initialized, Role: FOLLOWER
I20260620 15:00:19.188125 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52: Bootstrap starting.
I20260620 15:00:19.188345 20904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.189639 20904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:19.190686 20828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:19.190858 20753 raft_consensus.cc:348] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.191088 20753 raft_consensus.cc:374] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:19.191148 20753 raft_consensus.cc:729] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33af4b7b3774f648f2d179a3136554a, State: Initialized, Role: FOLLOWER
I20260620 15:00:19.191632 20753 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.191951 20828 log.cc:824] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:19.193063 20753 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:19.193955 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52: No bootstrap required, opened a new log
W20260620 15:00:19.194787 20931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:00:19.194849 20931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:00:19.195133 20920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.195243 20920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 15:00:19.195343 20706 tablet.cc:2369] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:19.193090 20835 master_runner.cc:186] Error getting master registration for 127.7.46.190:43895: OK, instance_id { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" instance_seqno: 1781967618699123 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:00:19.196475 20835 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:00:19.198767 20828 raft_consensus.cc:348] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.199061 20828 raft_consensus.cc:374] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:19.199163 20828 raft_consensus.cc:729] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64a31b7fe6c4a3e9b56dad711d82b52, State: Initialized, Role: FOLLOWER
I20260620 15:00:19.199661 20828 consensus_queue.cc:260] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.200934 20828 sys_catalog.cc:564] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:00:19.202545 20945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:00:19.202621 20945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:00:19.202714 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.185584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.7.46.189:36161)
0620 15:00:19.185677 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.185681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.186934 (+ 1253us) server_negotiation.cc:407] Connection header received
0620 15:00:19.187106 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.187109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.187151 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.187198 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.188341 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.189077 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.196782 (+ 7705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.197120 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.197193 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.199096 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.199100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.199103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.199128 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.200857 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.201324 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.201327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.201329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.201420 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.201711 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.201714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.201716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.201911 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.202006 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.202422 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.202557 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:00:19.203224 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.203351 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:19.200489 20685 master_runner.cc:186] Error getting master registration for 127.7.46.189:36161: OK, instance_id { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" instance_seqno: 1781967618913405 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:00:19.205596 20685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 15:00:19.206611 20958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:00:19.206684 20958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:00:19.210532 20760 master_runner.cc:263] No leader master found from master c64a31b7fe6c4a3e9b56dad711d82b52
I20260620 15:00:19.210603 20760 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:00:19.216143 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.185163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.7.46.190:43895)
0620 15:00:19.185265 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.185269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.188271 (+ 3002us) server_negotiation.cc:407] Connection header received
0620 15:00:19.190108 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.190112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.190158 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.190206 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.196737 (+ 6531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.197436 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.209143 (+ 11707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.209474 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.209573 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.212272 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.212276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.212279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.212305 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.213984 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.214464 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.214468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.214469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.214551 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.214831 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.214834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.214836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.215063 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.215196 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.215855 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.215982 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 15:00:19.216533 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.216719 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:19.221408 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:19.186404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.7.46.188:34551)
0620 15:00:19.186493 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:19.186497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:19.190036 (+ 3539us) server_negotiation.cc:407] Connection header received
0620 15:00:19.190978 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:19.190980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:19.191025 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:19.191076 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:19.198107 (+ 7031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.198937 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.206700 (+ 7763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:19.207072 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:19.207148 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:19.217715 (+ 10567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:19.217720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:19.217722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:19.217752 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:19.219500 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.220023 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.220026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.220029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.220115 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:19.220410 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:19.220414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:19.220416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:19.220642 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:19.220758 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:19.221124 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 15:00:19.221254 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 15:00:19.371178 20946 raft_consensus.cc:482] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:19.371407 20946 raft_consensus.cc:504] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.372449 20946 leader_election.cc:288] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d33af4b7b3774f648f2d179a3136554a (127.7.46.190:43895), 66c42bbfe0044a48ae6c05bfa4f40e23 (127.7.46.188:34551)
I20260620 15:00:19.373019 20727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33af4b7b3774f648f2d179a3136554a" is_pre_election: true
I20260620 15:00:19.373350 20727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64a31b7fe6c4a3e9b56dad711d82b52 in term 0.
I20260620 15:00:19.373296 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" is_pre_election: true
I20260620 15:00:19.373585 20878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64a31b7fe6c4a3e9b56dad711d82b52 in term 0.
I20260620 15:00:19.374001 20778 leader_election.cc:302] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64a31b7fe6c4a3e9b56dad711d82b52, d33af4b7b3774f648f2d179a3136554a; no voters:
I20260620 15:00:19.374310 20946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:19.374404 20946 raft_consensus.cc:482] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:19.374449 20946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:19.375681 20946 raft_consensus.cc:504] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.376163 20946 leader_election.cc:288] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [CANDIDATE]: Term 1 election: Requested vote from peers d33af4b7b3774f648f2d179a3136554a (127.7.46.190:43895), 66c42bbfe0044a48ae6c05bfa4f40e23 (127.7.46.188:34551)
W20260620 15:00:19.376287 20857 tablet.cc:2369] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:19.376681 20727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33af4b7b3774f648f2d179a3136554a"
I20260620 15:00:19.376796 20727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:19.377470 20878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23"
I20260620 15:00:19.377578 20878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:19.378080 20727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64a31b7fe6c4a3e9b56dad711d82b52 in term 1.
I20260620 15:00:19.378356 20778 leader_election.cc:302] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64a31b7fe6c4a3e9b56dad711d82b52, d33af4b7b3774f648f2d179a3136554a; no voters:
I20260620 15:00:19.378509 20946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:19.378732 20946 raft_consensus.cc:686] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [term 1 LEADER]: Becoming Leader. State: Replica: c64a31b7fe6c4a3e9b56dad711d82b52, State: Running, Role: LEADER
I20260620 15:00:19.378830 20878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64a31b7fe6c4a3e9b56dad711d82b52 in term 1.
I20260620 15:00:19.379076 20946 consensus_queue.cc:237] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } }
I20260620 15:00:19.381675 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c64a31b7fe6c4a3e9b56dad711d82b52. Latest consensus state: current_term: 1 leader_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.381803 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:19.382429 20962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:19.383520 20962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:19.387907 20878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [term 1 FOLLOWER]: Refusing update from remote peer c64a31b7fe6c4a3e9b56dad711d82b52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:19.387993 20727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [term 1 FOLLOWER]: Refusing update from remote peer c64a31b7fe6c4a3e9b56dad711d82b52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:19.388579 20946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:19.388713 20960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:19.392198 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c64a31b7fe6c4a3e9b56dad711d82b52. Latest consensus state: current_term: 1 leader_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.392314 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:19.392748 20920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c64a31b7fe6c4a3e9b56dad711d82b52. Latest consensus state: current_term: 1 leader_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.392869 20920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:19.394994 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.395093 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:19.396076 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.396174 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:19.396486 20964 mvcc.cc:205] Tried to move back new op lower bound from 7298939369006645248 to 7298939368980406272. Current Snapshot: MvccSnapshot[applied={T|T < 7298939369006645248}]
I20260620 15:00:19.396632 20963 mvcc.cc:205] Tried to move back new op lower bound from 7298939369006645248 to 7298939368980406272. Current Snapshot: MvccSnapshot[applied={T|T < 7298939369006645248}]
I20260620 15:00:19.398120 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.398218 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:19.399168 20920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d33af4b7b3774f648f2d179a3136554a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43895 } } peers { permanent_uuid: "c64a31b7fe6c4a3e9b56dad711d82b52" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 36161 } } peers { permanent_uuid: "66c42bbfe0044a48ae6c05bfa4f40e23" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 34551 } } }
I20260620 15:00:19.399276 20920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:19.403393 20962 catalog_manager.cc:1347] Generated new cluster ID: 58c6b179fdd94a809802ee5693b2e7dc
I20260620 15:00:19.403450 20962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:19.423350 20962 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:19.424536 20962 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:19.430390 20962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c64a31b7fe6c4a3e9b56dad711d82b52: Generated new TSK 0
I20260620 15:00:19.431205 20962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:00:20.196128 20931 catalog_manager.cc:1259] Loaded cluster ID: 58c6b179fdd94a809802ee5693b2e7dc
I20260620 15:00:20.196225 20931 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: loading cluster ID for follower catalog manager: success
I20260620 15:00:20.197831 20931 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: acquiring CA information for follower catalog manager: success
I20260620 15:00:20.198921 20931 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66c42bbfe0044a48ae6c05bfa4f40e23: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:00:20.203687 20945 catalog_manager.cc:1259] Loaded cluster ID: 58c6b179fdd94a809802ee5693b2e7dc
I20260620 15:00:20.203760 20945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: loading cluster ID for follower catalog manager: success
I20260620 15:00:20.205327 20945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: acquiring CA information for follower catalog manager: success
I20260620 15:00:20.206441 20945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d33af4b7b3774f648f2d179a3136554a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:00:20.493427 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 20685
I20260620 15:00:20.500087 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 20760
I20260620 15:00:20.506062 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 20835
2026-06-20T15:00:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3139 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:20 dist-test-slave-30t4 krb5kdc[20977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:20 dist-test-slave-30t4 krb5kdc[20977](info): set up 2 sockets
Jun 20 15:00:20 dist-test-slave-30t4 krb5kdc[20977](info): commencing operation
krb5kdc: starting...
W20260620 15:00:21.548069 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:00:21 dist-test-slave-30t4 krb5kdc[20977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:00:21.666880 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:38517
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:38517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:21.776798 20990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:21.777041 20990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:21.777084 20990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:21.780884 20990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:21.780936 20990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:21.780957 20990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:21.780977 20990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:21.780994 20990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:21.786090 20990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:38517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:38517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:21.786986 20990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:21.787943 20990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:21.796319 20995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:21.796437 20998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:21.796322 20996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:21.796568 20990 server_base.cc:1034] running on GCE node
I20260620 15:00:21.797210 20990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:21.798204 20990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:21.799353 20990 hybrid_clock.cc:648] HybridClock initialized: now 1781967621799335 us; error 30 us; skew 500 ppm
Jun 20 15:00:21 dist-test-slave-30t4 krb5kdc[20977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:21.802095 20990 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:00:21.802927 20990 webserver.cc:460] Webserver started at http://127.7.46.190:36477/ using document root <none> and password file <none>
I20260620 15:00:21.803460 20990 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:21.803555 20990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:21.803822 20990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:21.805846 20990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-30t4"
server_key: "500002ac6718ebc286ef6635f047fa86"
server_key_iv: "7a67340ee263ff882bfbadda58d894cb"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.806454 20990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-30t4"
server_key: "500002ac6718ebc286ef6635f047fa86"
server_key_iv: "7a67340ee263ff882bfbadda58d894cb"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.810086 20990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 15:00:21.812566 21005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:21.813897 20990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:00:21.814060 20990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-30t4"
server_key: "500002ac6718ebc286ef6635f047fa86"
server_key_iv: "7a67340ee263ff882bfbadda58d894cb"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.814172 20990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:21.843204 20990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:21.846190 20990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:21.846390 20990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:21.853156 20990 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:38517
I20260620 15:00:21.853188 21057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:38517 every 8 connection(s)
I20260620 15:00:21.854074 20990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:21.856704 21058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:21.862051 21058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b: Bootstrap starting.
I20260620 15:00:21.862501 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 20990
I20260620 15:00:21.862635 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:00:21.862920 7354 external_mini_cluster.cc:1428] Setting key 7a2a28864d32c1e8acc54c1fda6dd0ac
I20260620 15:00:21.864727 21058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:21.865446 21058 log.cc:824] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:21.867220 21058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b: No bootstrap required, opened a new log
I20260620 15:00:21.870709 21058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38517 } }
I20260620 15:00:21.870983 21058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:21.871034 21058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe1d4e2a90d4eeb9e7467e8d34bf55b, State: Initialized, Role: FOLLOWER
Jun 20 15:00:21 dist-test-slave-30t4 krb5kdc[20977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:21.871516 21058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38517 } }
I20260620 15:00:21.871680 21058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:21.871757 21058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:21.871888 21058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:21.872943 21058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38517 } }
I20260620 15:00:21.873271 21058 leader_election.cc:302] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fe1d4e2a90d4eeb9e7467e8d34bf55b; no voters:
I20260620 15:00:21.873576 21058 leader_election.cc:288] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:21.873679 21063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:21.873909 21063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [term 1 LEADER]: Becoming Leader. State: Replica: 5fe1d4e2a90d4eeb9e7467e8d34bf55b, State: Running, Role: LEADER
I20260620 15:00:21.874284 21063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38517 } }
I20260620 15:00:21.874886 21058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:21.875924 21064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38517 } } }
I20260620 15:00:21.876068 21064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:21.876397 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe1d4e2a90d4eeb9e7467e8d34bf55b. Latest consensus state: current_term: 1 leader_uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe1d4e2a90d4eeb9e7467e8d34bf55b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38517 } } }
I20260620 15:00:21.876497 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:21.878042 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.864202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.7.46.190:38517)
0620 15:00:21.864626 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:21.864640 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:21.864674 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:00:21.865339 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:21.865355 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:21.865728 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:21.866048 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:21.866854 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.867919 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.868764 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.869236 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.869408 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.872120 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:21.872125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:21.872136 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:21.872162 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:21.874129 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.874654 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.874659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.874665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.874799 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.875109 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.875112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.875114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.875639 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:21.876732 (+ 1093us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:21.876876 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 15:00:21.877352 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
W20260620 15:00:21.880607 21078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:00:21.880715 21078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:00:21.880788 21072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:21.880781 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:00:21.881649 21072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:21.886749 21072 catalog_manager.cc:1347] Generated new cluster ID: 0ebf2de0d23d44d38440e18727d1695d
I20260620 15:00:21.886816 21072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:21.916821 21072 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:21.917904 21072 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:21.923521 21072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fe1d4e2a90d4eeb9e7467e8d34bf55b: Generated new TSK 0
I20260620 15:00:21.924296 21072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:00:21.936249 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 20990
2026-06-20T15:00:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1428 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:21 dist-test-slave-30t4 krb5kdc[21085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:21 dist-test-slave-30t4 krb5kdc[21085](info): set up 2 sockets
Jun 20 15:00:21 dist-test-slave-30t4 krb5kdc[21085](info): commencing operation
krb5kdc: starting...
W20260620 15:00:23.008085 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:00:23.124408 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44819
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:44819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:23.230387 21098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:23.230615 21098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:23.230674 21098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:23.234558 21098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:23.234607 21098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:23.234627 21098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:23.234683 21098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:23.234719 21098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:23.239749 21098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:44819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:23.240650 21098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:23.241568 21098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:23.249184 21103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.249257 21104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.249184 21106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:23.249367 21098 server_base.cc:1034] running on GCE node
I20260620 15:00:23.249931 21098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:23.250928 21098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:23.252099 21098 hybrid_clock.cc:648] HybridClock initialized: now 1781967623252069 us; error 45 us; skew 500 ppm
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:23.254670 21098 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:00:23.255499 21098 webserver.cc:460] Webserver started at http://127.7.46.190:45039/ using document root <none> and password file <none>
I20260620 15:00:23.256062 21098 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:23.256145 21098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:23.256347 21098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:23.258231 21098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "74111f51ba5643fdb56dccd2e93be8a5"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "6763372c23b88710db39814da18863a1"
server_key_iv: "541424056b03f9221fc71ed5b7a3f172"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.258775 21098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "74111f51ba5643fdb56dccd2e93be8a5"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "6763372c23b88710db39814da18863a1"
server_key_iv: "541424056b03f9221fc71ed5b7a3f172"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.262352 21098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 15:00:23.264760 21113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.265946 21098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:00:23.266083 21098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "74111f51ba5643fdb56dccd2e93be8a5"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "6763372c23b88710db39814da18863a1"
server_key_iv: "541424056b03f9221fc71ed5b7a3f172"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.266189 21098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:23.284569 21098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:23.287539 21098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:23.287747 21098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:23.294516 21098 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:44819
I20260620 15:00:23.294880 21165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:44819 every 8 connection(s)
I20260620 15:00:23.295415 21098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:23.298074 21166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:23.299871 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 21098
I20260620 15:00:23.300020 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:00:23.300283 7354 external_mini_cluster.cc:1428] Setting key 4d491d060992ad3af113ab678ba2498b
I20260620 15:00:23.304266 21166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Bootstrap starting.
I20260620 15:00:23.306613 21166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:23.307384 21166 log.cc:824] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:23.309136 21166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: No bootstrap required, opened a new log
I20260620 15:00:23.312582 21166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:23.312803 21166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:23.312870 21166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74111f51ba5643fdb56dccd2e93be8a5, State: Initialized, Role: FOLLOWER
I20260620 15:00:23.313356 21166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:23.313516 21166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:23.313587 21166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:23.313680 21166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:23.314556 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:23.301495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.7.46.190:44819)
0620 15:00:23.301851 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:23.301868 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:23.301909 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:00:23.302513 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:23.302529 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:23.302824 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:23.303074 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:23.303938 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.304923 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.306099 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.306621 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.306825 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:23.309498 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:23.309504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:23.309517 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:23.309544 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:23.311208 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.311677 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.311688 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.311692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.311864 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.312161 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.312165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.312167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.312562 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:23.312783 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:23.313313 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 15:00:23.313626 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":110,"threads_started":1}
I20260620 15:00:23.314670 21166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:23.314985 21166 leader_election.cc:302] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74111f51ba5643fdb56dccd2e93be8a5; no voters:
I20260620 15:00:23.315253 21166 leader_election.cc:288] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:23.315368 21171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:23.315595 21171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 1 LEADER]: Becoming Leader. State: Replica: 74111f51ba5643fdb56dccd2e93be8a5, State: Running, Role: LEADER
I20260620 15:00:23.315994 21171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:23.316408 21166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:23.317772 21172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74111f51ba5643fdb56dccd2e93be8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } } }
I20260620 15:00:23.317917 21172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:23.318275 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74111f51ba5643fdb56dccd2e93be8a5. Latest consensus state: current_term: 1 leader_uuid: "74111f51ba5643fdb56dccd2e93be8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } } }
I20260620 15:00:23.318364 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: This master's current role is: LEADER
W20260620 15:00:23.321230 21186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:00:23.321298 21186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:00:23.321376 21182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:23.322039 21182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:23.327153 21182 catalog_manager.cc:1347] Generated new cluster ID: 6f35081b35544658aa4bdb2e226a1808
I20260620 15:00:23.327214 21182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:23.342280 21182 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:23.343117 21182 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:23.347946 21182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Generated new TSK 0
I20260620 15:00:23.348556 21182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 15:00:23.379640 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44819
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:23.487020 21192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:23.487263 21192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:23.487305 21192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:23.491359 21192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:23.491412 21192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:23.491468 21192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:00:23.496464 21192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:23.497359 21192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:23.498267 21192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:23.507323 21200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.507360 21197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.507333 21198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:23.507521 21192 server_base.cc:1034] running on GCE node
I20260620 15:00:23.508010 21192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:23.508592 21192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:23.509753 21192 hybrid_clock.cc:648] HybridClock initialized: now 1781967623509737 us; error 32 us; skew 500 ppm
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:23.512445 21192 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:00:23.513250 21192 webserver.cc:460] Webserver started at http://127.7.46.129:44001/ using document root <none> and password file <none>
I20260620 15:00:23.513785 21192 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:23.513868 21192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:23.514074 21192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:23.516036 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "d5ded6fb27b242ab93da71151dffdacf"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "c444a8a9692e2b57856fe6f928e5ce2c"
server_key_iv: "461cc7702707f030fc4c2d5316a4be60"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.516595 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "d5ded6fb27b242ab93da71151dffdacf"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "c444a8a9692e2b57856fe6f928e5ce2c"
server_key_iv: "461cc7702707f030fc4c2d5316a4be60"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.520285 21192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:23.522552 21207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.523669 21192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:23.523824 21192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "d5ded6fb27b242ab93da71151dffdacf"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "c444a8a9692e2b57856fe6f928e5ce2c"
server_key_iv: "461cc7702707f030fc4c2d5316a4be60"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.523927 21192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:23.545619 21192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:23.548650 21192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:23.548833 21192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:23.549412 21192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:23.550196 21192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:23.550274 21192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.550354 21192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:23.550400 21192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.559088 21192 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:42405
I20260620 15:00:23.559108 21320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:42405 every 8 connection(s)
I20260620 15:00:23.560010 21192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:23.565423 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 21192
I20260620 15:00:23.565568 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:23.565835 7354 external_mini_cluster.cc:1428] Setting key ee6e82834304017daf45ccd302cfe406
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:23.575107 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:23.561862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:49583 (local address 127.7.46.190:44819)
0620 15:00:23.562009 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:23.562013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:23.563254 (+ 1241us) server_negotiation.cc:407] Connection header received
0620 15:00:23.564186 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:23.564189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:23.564244 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:23.564345 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:23.565605 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.566246 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.567669 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.567973 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.568057 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:23.571156 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:23.571159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:23.571161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:23.571183 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:23.573058 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.573552 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.573557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.573561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.573663 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.573973 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.573980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.573982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.574230 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:23.574333 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:23.574783 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 15:00:23.574913 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:00:23.575852 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:23.562228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:49583)
0620 15:00:23.563100 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:23.563141 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:23.563988 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:23.564511 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:23.564517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:23.564849 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:23.565430 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:23.565451 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.566391 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.566395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:23.567553 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:23.567564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.568117 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.568124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:23.568352 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:23.569207 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:23.569222 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:23.571003 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:23.573176 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:23.573181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:23.573191 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:23.573446 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:23.573753 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:23.573755 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:23.573757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:23.573882 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:23.574336 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:23.574341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:23.574554 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 15:00:23.574706 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:00:23.574943 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":537,"threads_started":1}
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 15:00:23.577003 21321 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44819
I20260620 15:00:23.577230 21321 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:23.577682 21321 heartbeater.cc:511] Master 127.7.46.190:44819 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 15:00:23.579166 21130 ts_manager.cc:194] Registered new tserver with Master: d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405)
I20260620 15:00:23.580318 21130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:49583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 15:00:23.595150 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44819
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:23.701026 21326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:23.701296 21326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:23.701368 21326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:23.705492 21326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:23.705571 21326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:23.705653 21326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:00:23.710666 21326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:23.711598 21326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:23.712585 21326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:23.721843 21331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.721841 21332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.721879 21334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:23.722019 21326 server_base.cc:1034] running on GCE node
I20260620 15:00:23.722554 21326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:23.723276 21326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:23.724485 21326 hybrid_clock.cc:648] HybridClock initialized: now 1781967623724455 us; error 46 us; skew 500 ppm
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:23.728250 21326 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:00:23.729116 21326 webserver.cc:460] Webserver started at http://127.7.46.130:35673/ using document root <none> and password file <none>
I20260620 15:00:23.729609 21326 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:23.729664 21326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:23.729815 21326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:23.731664 21326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "827c0252f33646deb8ceea442c97b632"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "93ab1c1f39bbe1eb6def67d7005c13f7"
server_key_iv: "9917db42a74401be9e0e622e0a629b34"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.732221 21326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "827c0252f33646deb8ceea442c97b632"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "93ab1c1f39bbe1eb6def67d7005c13f7"
server_key_iv: "9917db42a74401be9e0e622e0a629b34"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.735688 21326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:23.738041 21341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.739228 21326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:23.739351 21326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "827c0252f33646deb8ceea442c97b632"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "93ab1c1f39bbe1eb6def67d7005c13f7"
server_key_iv: "9917db42a74401be9e0e622e0a629b34"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.739457 21326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:23.763661 21326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:23.766993 21326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:23.767230 21326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:23.767902 21326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:23.768810 21326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:23.768867 21326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.768915 21326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:23.768978 21326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.778939 21326 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:33553
I20260620 15:00:23.779007 21454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:33553 every 8 connection(s)
I20260620 15:00:23.779999 21326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:23.781111 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 21326
I20260620 15:00:23.781255 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:23.781590 7354 external_mini_cluster.cc:1428] Setting key b98136351391cbc147c54dfd2a7639dd
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
I20260620 15:00:23.794467 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:23.782864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:59841 (local address 127.7.46.190:44819)
0620 15:00:23.783060 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:23.783067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:23.783087 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:00:23.784280 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:23.784284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:23.784346 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:23.784459 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:23.785683 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.786373 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.787508 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.788017 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.788176 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:23.790571 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:23.790577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:23.790585 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:23.790620 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:23.792454 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.792973 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.792981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.792983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.793084 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.793380 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.793383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.793384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.793665 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:23.793801 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:23.794184 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 15:00:23.794298 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:00:23.795239 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:23.782107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:59841)
0620 15:00:23.782569 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:23.782616 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:23.784048 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:23.784633 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:23.784640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:23.785025 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:23.785524 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:23.785538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.786501 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.786503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:23.787347 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:23.787356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.788189 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.788192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:23.788329 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:23.788882 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:23.788904 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:23.790390 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:23.792578 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:23.792582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:23.792592 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:23.792868 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:23.793175 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:23.793177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:23.793178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:23.793298 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:23.793755 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:23.793760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:23.793973 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 15:00:23.794121 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 15:00:23.794346 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
I20260620 15:00:23.796370 21455 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44819
I20260620 15:00:23.796603 21455 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:23.797158 21455 heartbeater.cc:511] Master 127.7.46.190:44819 requested a full tablet report, sending...
I20260620 15:00:23.798309 21130 ts_manager.cc:194] Registered new tserver with Master: 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553)
I20260620 15:00:23.798875 21130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:59841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 15:00:23.812261 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44819
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:23.920105 21460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:23.920321 21460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:23.920361 21460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:23.924161 21460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:23.924206 21460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:23.924259 21460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:00:23.929388 21460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:23.930259 21460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:23.931139 21460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:23.940212 21466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:23.940279 21460 server_base.cc:1034] running on GCE node
W20260620 15:00:23.940294 21468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.940234 21465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:23.940835 21460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:23.941450 21460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:23.942649 21460 hybrid_clock.cc:648] HybridClock initialized: now 1781967623942630 us; error 29 us; skew 500 ppm
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:23.945420 21460 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:00:23.946180 21460 webserver.cc:460] Webserver started at http://127.7.46.131:32973/ using document root <none> and password file <none>
I20260620 15:00:23.946660 21460 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:23.946714 21460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:23.946872 21460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:23.948784 21460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "290e882a16c04470a10d2de37109584c"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "2dc8176fa06f741f71c82e0178910a1c"
server_key_iv: "745e76f069daf18fa8111f43ac6e8f43"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.949277 21460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "290e882a16c04470a10d2de37109584c"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "2dc8176fa06f741f71c82e0178910a1c"
server_key_iv: "745e76f069daf18fa8111f43ac6e8f43"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.952986 21460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:00:23.955279 21475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.956593 21460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:00:23.956713 21460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "290e882a16c04470a10d2de37109584c"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "2dc8176fa06f741f71c82e0178910a1c"
server_key_iv: "745e76f069daf18fa8111f43ac6e8f43"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.956815 21460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:23.966782 21460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:23.969935 21460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:23.970115 21460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:23.970669 21460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:23.971426 21460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:23.971469 21460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.971531 21460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:23.971565 21460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.979751 21460 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:36067
I20260620 15:00:23.979811 21588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:36067 every 8 connection(s)
I20260620 15:00:23.980599 21460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:23.988349 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 21460
I20260620 15:00:23.988483 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:23.988765 7354 external_mini_cluster.cc:1428] Setting key 07e23d458a455e355be2042b52bb2036
Jun 20 15:00:23 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:23.993590 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:23.982214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:41103 (local address 127.7.46.190:44819)
0620 15:00:23.982338 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:23.982342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:23.983056 (+ 714us) server_negotiation.cc:407] Connection header received
0620 15:00:23.984329 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:23.984332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:23.984392 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:23.984487 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:23.985577 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.986112 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.987214 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.987436 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.987539 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:23.990250 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:23.990253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:23.990255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:23.990279 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:23.991629 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.992149 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.992154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.992156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.992238 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.992564 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.992567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.992569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.992743 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:23.992843 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:23.993296 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 15:00:23.993414 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:00:23.994311 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:23.982526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:41103)
0620 15:00:23.982922 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:23.982962 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:23.984131 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:23.984646 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:23.984653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:23.984969 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:23.985443 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:23.985457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.986230 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.986238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:23.987086 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:23.987095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.987573 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.987577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:23.987733 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:23.988323 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:23.988346 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:23.990095 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:23.991739 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:23.991745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:23.991759 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:23.992048 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:23.992325 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:23.992329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:23.992331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:23.992466 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:23.992840 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:23.992846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:23.993071 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 15:00:23.993240 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 15:00:23.993557 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260620 15:00:23.995276 21589 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44819
I20260620 15:00:23.995479 21589 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:23.995946 21589 heartbeater.cc:511] Master 127.7.46.190:44819 requested a full tablet report, sending...
I20260620 15:00:23.996832 21130 ts_manager.cc:194] Registered new tserver with Master: 290e882a16c04470a10d2de37109584c (127.7.46.131:36067)
I20260620 15:00:23.997361 21130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:41103
I20260620 15:00:24.002617 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:24.011171 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.004492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.7.46.190:44819)
0620 15:00:24.004613 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.004616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.004739 (+ 123us) server_negotiation.cc:407] Connection header received
0620 15:00:24.004842 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.004844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.004901 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.004966 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:24.005582 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.006107 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.006885 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.007091 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.007166 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.008436 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:24.008439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:24.008441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:24.008463 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:24.009912 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.010236 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.010239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.010240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.010307 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.010524 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.010526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.010528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.010727 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:24.010826 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.010978 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.011061 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:00:24.013857 21130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:24.016899 21130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:24.031054 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.025598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:36067 (local address 127.0.0.1:54182)
0620 15:00:24.025802 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.025814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.025897 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.027057 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.027060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.027076 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:24.027169 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.027176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.028312 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.028314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.029592 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.029601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.030603 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.030606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.030679 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.030708 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.030819 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.030937 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":153,"thread_start_us":90,"threads_started":1}
I20260620 15:00:24.031100 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.025751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.7.46.131:36067)
0620 15:00:24.026590 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.026603 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.026625 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:00:24.026699 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.026722 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.026904 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.027079 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:24.027282 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.028113 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.029708 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.030481 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.030593 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.030730 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.030852 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.030958 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":167,"threads_started":1}
I20260620 15:00:24.032150 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.024936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42405 (local address 127.0.0.1:35656)
0620 15:00:24.025363 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.025375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.025596 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.026767 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.026770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.026788 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:24.026886 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.026892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.028283 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.028286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.029832 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.029843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.031720 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.031724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.031853 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.031876 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.031950 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.032020 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":157,"thread_start_us":87,"threads_started":1}
I20260620 15:00:24.032295 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.024896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:33553 (local address 127.0.0.1:43374)
0620 15:00:24.025301 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.025349 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.025489 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.026284 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.026291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.026317 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:24.026461 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.026468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.028209 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.028212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.029832 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.029843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.031919 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.031923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.032016 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.032042 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.032093 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.032166 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":265,"thread_start_us":67,"threads_started":1}
I20260620 15:00:24.032691 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.025989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.7.46.129:42405)
0620 15:00:24.026287 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.026300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.026321 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:00:24.026387 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.026392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.026583 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.026758 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:24.026997 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.028124 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.030873 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.031610 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.031695 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.031837 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.032469 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.032563 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260620 15:00:24.033084 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.025012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.7.46.130:33553)
0620 15:00:24.025774 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.025786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.025809 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:00:24.025890 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.025896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.026093 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.026610 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:24.026781 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.028032 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.029955 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.030760 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.032725 (+ 1965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.032871 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.032918 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.032981 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":116,"threads_started":1}
I20260620 15:00:24.034219 21523 tablet_service.cc:1467] Processing CreateTablet for tablet 178d4b9a152346439b3298f128092d9a (DEFAULT_TABLE table=test-table [id=34a27b28626345dcb5cb06181f20635f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:24.035351 21523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178d4b9a152346439b3298f128092d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:24.035077 21255 tablet_service.cc:1467] Processing CreateTablet for tablet 178d4b9a152346439b3298f128092d9a (DEFAULT_TABLE table=test-table [id=34a27b28626345dcb5cb06181f20635f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:24.035983 21255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178d4b9a152346439b3298f128092d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:24.035575 21389 tablet_service.cc:1467] Processing CreateTablet for tablet 178d4b9a152346439b3298f128092d9a (DEFAULT_TABLE table=test-table [id=34a27b28626345dcb5cb06181f20635f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:24.036497 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178d4b9a152346439b3298f128092d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:24.041735 21606 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Bootstrap starting.
I20260620 15:00:24.043094 21607 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Bootstrap starting.
I20260620 15:00:24.044163 21606 tablet_bootstrap.cc:654] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:24.045167 21607 tablet_bootstrap.cc:654] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:24.045470 21606 log.cc:824] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:24.046134 21607 log.cc:824] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:24.046672 21608 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: Bootstrap starting.
I20260620 15:00:24.047382 21606 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: No bootstrap required, opened a new log
I20260620 15:00:24.047573 21606 ts_tablet_manager.cc:1397] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 15:00:24.047986 21607 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: No bootstrap required, opened a new log
I20260620 15:00:24.048161 21607 ts_tablet_manager.cc:1397] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:24.048780 21608 tablet_bootstrap.cc:654] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:24.049383 21608 log.cc:824] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:24.051090 21608 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: No bootstrap required, opened a new log
I20260620 15:00:24.051268 21608 ts_tablet_manager.cc:1397] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:00:24.051424 21606 raft_consensus.cc:348] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.051682 21606 raft_consensus.cc:374] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:24.051780 21606 raft_consensus.cc:729] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ded6fb27b242ab93da71151dffdacf, State: Initialized, Role: FOLLOWER
I20260620 15:00:24.051818 21607 raft_consensus.cc:348] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.052142 21607 raft_consensus.cc:374] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:24.052205 21607 raft_consensus.cc:729] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290e882a16c04470a10d2de37109584c, State: Initialized, Role: FOLLOWER
I20260620 15:00:24.052251 21606 consensus_queue.cc:260] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.052903 21321 heartbeater.cc:503] Master 127.7.46.190:44819 was elected leader, sending a full tablet report...
I20260620 15:00:24.053236 21606 ts_tablet_manager.cc:1428] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 15:00:24.052692 21607 consensus_queue.cc:260] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.053978 21589 heartbeater.cc:503] Master 127.7.46.190:44819 was elected leader, sending a full tablet report...
I20260620 15:00:24.054267 21607 ts_tablet_manager.cc:1428] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 15:00:24.055814 21608 raft_consensus.cc:348] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.056074 21608 raft_consensus.cc:374] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:24.056130 21608 raft_consensus.cc:729] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827c0252f33646deb8ceea442c97b632, State: Initialized, Role: FOLLOWER
I20260620 15:00:24.056515 21608 consensus_queue.cc:260] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.057415 21608 ts_tablet_manager.cc:1428] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 15:00:24.057600 21455 heartbeater.cc:503] Master 127.7.46.190:44819 was elected leader, sending a full tablet report...
W20260620 15:00:24.061149 21322 tablet.cc:2369] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:24.180364 21613 raft_consensus.cc:482] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:24.180621 21613 raft_consensus.cc:504] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.181732 21613 leader_election.cc:288] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553), d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405)
I20260620 15:00:24.184993 21614 raft_consensus.cc:482] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:24.185204 21614 raft_consensus.cc:504] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.185406 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.181933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:33553 (local address 127.7.46.131:38899)
0620 15:00:24.182103 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.182119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.182209 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.182491 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.182494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.182511 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:24.182628 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.182634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.183331 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.183337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.184268 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.184277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.184988 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.184992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.185100 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.185122 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.185188 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.185263 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":1}
I20260620 15:00:24.185436 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.182064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:38899 (local address 127.7.46.130:33553)
0620 15:00:24.182185 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.182192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.182211 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:00:24.182344 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.182346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.182393 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.182481 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:24.182735 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.183225 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.184387 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.184862 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.184978 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.185032 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.185237 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.185313 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:00:24.185528 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.182135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:47093 (local address 127.7.46.129:42405)
0620 15:00:24.182320 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.182326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.182397 (+ 71us) server_negotiation.cc:407] Connection header received
0620 15:00:24.182548 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.182551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.182600 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.182682 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:24.182922 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.183398 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.184484 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.184859 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.185001 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.185057 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.185328 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.185395 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 15:00:24.185884 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.181974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42405 (local address 127.7.46.131:47093)
0620 15:00:24.182304 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.182318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.182442 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.182692 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.182694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.182710 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:24.182819 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.182824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.183515 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.183518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.184355 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.184364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.185100 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.185103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.185201 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.185219 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.185657 (+ 438us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.185741 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
I20260620 15:00:24.186260 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178d4b9a152346439b3298f128092d9a" candidate_uuid: "290e882a16c04470a10d2de37109584c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827c0252f33646deb8ceea442c97b632" is_pre_election: true
I20260620 15:00:24.186471 21409 raft_consensus.cc:2454] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290e882a16c04470a10d2de37109584c in term 0.
I20260620 15:00:24.186590 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178d4b9a152346439b3298f128092d9a" candidate_uuid: "290e882a16c04470a10d2de37109584c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ded6fb27b242ab93da71151dffdacf" is_pre_election: true
I20260620 15:00:24.186867 21275 raft_consensus.cc:2454] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290e882a16c04470a10d2de37109584c in term 0.
I20260620 15:00:24.186901 21614 leader_election.cc:288] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405), 290e882a16c04470a10d2de37109584c (127.7.46.131:36067)
I20260620 15:00:24.187100 21479 leader_election.cc:302] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632; no voters:
I20260620 15:00:24.187621 21613 raft_consensus.cc:2781] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:24.187714 21613 raft_consensus.cc:482] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:24.187788 21613 raft_consensus.cc:3037] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:24.189064 21613 raft_consensus.cc:504] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.189415 21613 leader_election.cc:288] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [CANDIDATE]: Term 1 election: Requested vote from peers 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553), d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405)
I20260620 15:00:24.189733 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178d4b9a152346439b3298f128092d9a" candidate_uuid: "290e882a16c04470a10d2de37109584c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827c0252f33646deb8ceea442c97b632"
I20260620 15:00:24.189801 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178d4b9a152346439b3298f128092d9a" candidate_uuid: "290e882a16c04470a10d2de37109584c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ded6fb27b242ab93da71151dffdacf"
I20260620 15:00:24.189947 21275 raft_consensus.cc:3037] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:24.189838 21409 raft_consensus.cc:3037] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:24.190668 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.186617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42405 (local address 127.7.46.130:51839)
0620 15:00:24.186766 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.186779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.186849 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.187535 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.187539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.187556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:24.187666 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.187672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.188508 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.188513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.189336 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.189345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.190298 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.190301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.190395 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.190416 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.190487 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.190558 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 15:00:24.190999 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.187107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:51839 (local address 127.7.46.129:42405)
0620 15:00:24.187237 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.187240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.187251 (+ 11us) server_negotiation.cc:407] Connection header received
0620 15:00:24.187283 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.187285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.187328 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.187406 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:24.187815 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.188400 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.189499 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.190058 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.190166 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.190237 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.190808 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.190876 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:00:24.191226 21409 raft_consensus.cc:2454] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290e882a16c04470a10d2de37109584c in term 1.
I20260620 15:00:24.191275 21275 raft_consensus.cc:2454] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290e882a16c04470a10d2de37109584c in term 1.
I20260620 15:00:24.191577 21479 leader_election.cc:302] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632; no voters:
I20260620 15:00:24.191706 21616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.187093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:36067 (local address 127.7.46.130:48209)
0620 15:00:24.187585 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.187597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.187682 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.187976 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.187979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.187997 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:24.188100 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.188105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.188991 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.188997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.190253 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.190265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.191249 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.191253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.191352 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.191378 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.191494 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.191572 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":123,"threads_started":1}
I20260620 15:00:24.191864 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.187449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:48209 (local address 127.7.46.131:36067)
0620 15:00:24.187571 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.187576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.187659 (+ 83us) server_negotiation.cc:407] Connection header received
0620 15:00:24.187817 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.187820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.187870 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.187937 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:24.188205 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.188857 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.190392 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.190972 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.191067 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.191135 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.191644 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.191713 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:00:24.191980 21613 raft_consensus.cc:2781] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:24.192062 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178d4b9a152346439b3298f128092d9a" candidate_uuid: "827c0252f33646deb8ceea442c97b632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ded6fb27b242ab93da71151dffdacf" is_pre_election: true
I20260620 15:00:24.192193 21613 raft_consensus.cc:686] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 1 LEADER]: Becoming Leader. State: Replica: 290e882a16c04470a10d2de37109584c, State: Running, Role: LEADER
I20260620 15:00:24.192241 21275 raft_consensus.cc:2379] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 827c0252f33646deb8ceea442c97b632 in current term 1: Already voted for candidate 290e882a16c04470a10d2de37109584c in this term.
I20260620 15:00:24.192294 21543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178d4b9a152346439b3298f128092d9a" candidate_uuid: "827c0252f33646deb8ceea442c97b632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290e882a16c04470a10d2de37109584c" is_pre_election: true
I20260620 15:00:24.192507 21613 consensus_queue.cc:237] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.192839 21342 leader_election.cc:302] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 827c0252f33646deb8ceea442c97b632; no voters: 290e882a16c04470a10d2de37109584c, d5ded6fb27b242ab93da71151dffdacf
I20260620 15:00:24.193081 21614 raft_consensus.cc:2726] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 15:00:24.195637 21129 catalog_manager.cc:5495] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c reported cstate change: term changed from 0 to 1, leader changed from <none> to 290e882a16c04470a10d2de37109584c (127.7.46.131). New cstate: current_term: 1 leader_uuid: "290e882a16c04470a10d2de37109584c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } health_report { overall_health: HEALTHY } } }
I20260620 15:00:24.221136 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.217963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.7.46.131:36067)
0620 15:00:24.218077 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.218080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.218092 (+ 12us) server_negotiation.cc:407] Connection header received
0620 15:00:24.218254 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.218260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.218328 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.218414 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:24.218600 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.219069 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.219898 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.220107 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.220184 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.220417 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:24.220794 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:24.220882 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.220979 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.221019 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 15:00:24.226519 21275 raft_consensus.cc:1261] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Refusing update from remote peer 290e882a16c04470a10d2de37109584c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:24.226629 21409 raft_consensus.cc:1261] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 1 FOLLOWER]: Refusing update from remote peer 290e882a16c04470a10d2de37109584c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:24.227164 21613 consensus_queue.cc:1035] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:24.227350 21617 consensus_queue.cc:1035] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [LEADER]: Connected to new peer: Peer: permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:00:24.231448 21590 tablet.cc:2369] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:24.234823 21622 mvcc.cc:205] Tried to move back new op lower bound from 7298939388825501696 to 7298939388694990848. Current Snapshot: MvccSnapshot[applied={T|T < 7298939388825501696}]
I20260620 15:00:24.236640 21621 mvcc.cc:205] Tried to move back new op lower bound from 7298939388825501696 to 7298939388694990848. Current Snapshot: MvccSnapshot[applied={T|T < 7298939388825501696}]
I20260620 15:00:24.241863 21129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:24.242015 21129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:24.244223 21129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Sending DeleteTablet for 3 replicas of tablet 178d4b9a152346439b3298f128092d9a
I20260620 15:00:24.244899 21389 tablet_service.cc:1514] Processing DeleteTablet for tablet 178d4b9a152346439b3298f128092d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:24 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:43374
I20260620 15:00:24.245126 21523 tablet_service.cc:1514] Processing DeleteTablet for tablet 178d4b9a152346439b3298f128092d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:24 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:54182
I20260620 15:00:24.245186 21255 tablet_service.cc:1514] Processing DeleteTablet for tablet 178d4b9a152346439b3298f128092d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:24 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:35656
I20260620 15:00:24.245524 21629 tablet_replica.cc:330] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: stopping tablet replica
I20260620 15:00:24.245523 21630 tablet_replica.cc:330] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: stopping tablet replica
I20260620 15:00:24.245743 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 21192
I20260620 15:00:24.245801 21629 raft_consensus.cc:2229] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:24.245800 21630 raft_consensus.cc:2229] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:24.246063 21629 raft_consensus.cc:2258] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:24.246093 21630 raft_consensus.cc:2258] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:24.247457 21629 ts_tablet_manager.cc:1905] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:24.247663 21630 ts_tablet_manager.cc:1905] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:24.250272 21630 ts_tablet_manager.cc:1918] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:24.250272 21629 ts_tablet_manager.cc:1918] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:24.250371 21629 log.cc:1196] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/178d4b9a152346439b3298f128092d9a
I20260620 15:00:24.250371 21630 log.cc:1196] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/178d4b9a152346439b3298f128092d9a
I20260620 15:00:24.250778 21630 ts_tablet_manager.cc:1939] T 178d4b9a152346439b3298f128092d9a P 290e882a16c04470a10d2de37109584c: Deleting consensus metadata
I20260620 15:00:24.251055 21629 ts_tablet_manager.cc:1939] T 178d4b9a152346439b3298f128092d9a P 827c0252f33646deb8ceea442c97b632: Deleting consensus metadata
I20260620 15:00:24.251964 21114 catalog_manager.cc:4841] TS 290e882a16c04470a10d2de37109584c (127.7.46.131:36067): tablet 178d4b9a152346439b3298f128092d9a (table test-table [id=34a27b28626345dcb5cb06181f20635f]) successfully deleted
I20260620 15:00:24.252074 21117 catalog_manager.cc:4841] TS 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553): tablet 178d4b9a152346439b3298f128092d9a (table test-table [id=34a27b28626345dcb5cb06181f20635f]) successfully deleted
I20260620 15:00:24.256165 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 21326
W20260620 15:00:24.257273 21114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:42405 (error 108)
I20260620 15:00:24.258283 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.257989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42405 (local address 127.0.0.1:35668)
0620 15:00:24.258110 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.258184 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:00:24.260324 21114 catalog_manager.cc:4584] TS d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 178d4b9a152346439b3298f128092d9a: Network error: Client connection negotiation failed: client connection to 127.7.46.129:42405: connect: Connection refused (error 111)
I20260620 15:00:24.262790 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 21460
I20260620 15:00:24.268407 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 21098
I20260620 15:00:24.274250 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44819
--webserver_interface=127.7.46.190
--webserver_port=45039
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:44819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:24.379149 21633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:24.379428 21633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:24.379498 21633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:24.383424 21633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:24.383503 21633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:24.383543 21633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:24.383594 21633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:24.383632 21633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:24.388749 21633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:44819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=45039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:24.389751 21633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:24.390797 21633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:24.403327 21640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.403242 21642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.403242 21639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:24.403695 21633 server_base.cc:1034] running on GCE node
I20260620 15:00:24.404275 21633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:24.405393 21633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:24.406560 21633 hybrid_clock.cc:648] HybridClock initialized: now 1781967624406540 us; error 28 us; skew 500 ppm
Jun 20 15:00:24 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:24.409356 21633 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:00:24.410255 21633 webserver.cc:460] Webserver started at http://127.7.46.190:45039/ using document root <none> and password file <none>
I20260620 15:00:24.410811 21633 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:24.410903 21633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:24.414747 21633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:24.416831 21649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.418004 21633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:00:24.418135 21633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "74111f51ba5643fdb56dccd2e93be8a5"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "6763372c23b88710db39814da18863a1"
server_key_iv: "541424056b03f9221fc71ed5b7a3f172"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.418731 21633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:24.434614 21633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:24.437444 21633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:24.437651 21633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:24.444687 21633 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:44819
I20260620 15:00:24.444962 21701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:44819 every 8 connection(s)
I20260620 15:00:24.445753 21633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:24.450062 21702 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:00:24.450066 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 21633
I20260620 15:00:24.450557 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:42405
--local_ip_for_outbound_sockets=127.7.46.129
--tserver_master_addrs=127.7.46.190:44819
--webserver_port=44001
--webserver_interface=127.7.46.129
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:24.452373 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Bootstrap starting.
I20260620 15:00:24.469166 21702 log.cc:824] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:24.477866 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:00:24.478228 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Bootstrap complete.
I20260620 15:00:24.482219 21702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:24.482599 21702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74111f51ba5643fdb56dccd2e93be8a5, State: Initialized, Role: FOLLOWER
I20260620 15:00:24.483054 21702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:24.483172 21702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:24.483263 21702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:24.483369 21702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:00:24.485260 21702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:24.485579 21702 leader_election.cc:302] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74111f51ba5643fdb56dccd2e93be8a5; no voters:
I20260620 15:00:24.485879 21702 leader_election.cc:288] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:00:24.486018 21706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:00:24.486284 21706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 2 LEADER]: Becoming Leader. State: Replica: 74111f51ba5643fdb56dccd2e93be8a5, State: Running, Role: LEADER
I20260620 15:00:24.486646 21706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:24.487039 21702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:24.488142 21708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74111f51ba5643fdb56dccd2e93be8a5. Latest consensus state: current_term: 2 leader_uuid: "74111f51ba5643fdb56dccd2e93be8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } } }
I20260620 15:00:24.488195 21707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74111f51ba5643fdb56dccd2e93be8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } } }
I20260620 15:00:24.488266 21708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:24.488299 21707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:24.488595 21715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:24.493558 21715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:24.494280 21715 catalog_manager.cc:1259] Loaded cluster ID: 6f35081b35544658aa4bdb2e226a1808
I20260620 15:00:24.494342 21715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:24.495759 21715 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 15:00:24.501088 21715 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 15:00:24.506624 21715 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:24.507381 21715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Loaded TSK: 0
I20260620 15:00:24.508628 21715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:00:24.571098 21704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:24.571337 21704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:24.571378 21704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:24.575374 21704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:24.575428 21704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:24.575505 21704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:00:24.580610 21704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:42405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=44001
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:24.581607 21704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:24.582546 21704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:24.591755 21729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:24.591886 21704 server_base.cc:1034] running on GCE node
W20260620 15:00:24.591755 21732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.592090 21730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:24.592515 21704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:24.593252 21704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:24.596539 21704 hybrid_clock.cc:648] HybridClock initialized: now 1781967624596506 us; error 48 us; skew 500 ppm
Jun 20 15:00:24 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:24.599291 21704 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:00:24.600245 21704 webserver.cc:460] Webserver started at http://127.7.46.129:44001/ using document root <none> and password file <none>
I20260620 15:00:24.600767 21704 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:24.600880 21704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:24.604712 21704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:24.606715 21739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.607946 21704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:00:24.608049 21704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "d5ded6fb27b242ab93da71151dffdacf"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "c444a8a9692e2b57856fe6f928e5ce2c"
server_key_iv: "461cc7702707f030fc4c2d5316a4be60"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.608542 21704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:24.639489 21704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:24.642534 21704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:24.642740 21704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:24.643344 21704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:24.644464 21746 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 15:00:24.648571 21704 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 15:00:24.648633 21704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 15:00:24.648705 21704 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 15:00:24.650334 21704 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 15:00:24.650377 21704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 15:00:24.650653 21746 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Bootstrap starting.
I20260620 15:00:24.661237 21704 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:42405
I20260620 15:00:24.661316 21853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:42405 every 8 connection(s)
I20260620 15:00:24.662180 21704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:24.667474 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 21704
I20260620 15:00:24.668068 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:33553
--local_ip_for_outbound_sockets=127.7.46.130
--tserver_master_addrs=127.7.46.190:44819
--webserver_port=35673
--webserver_interface=127.7.46.130
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:24.668821 21746 log.cc:824] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:00:24 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:24.676363 21746 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:00:24.676779 21746 tablet_bootstrap.cc:492] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Bootstrap complete.
I20260620 15:00:24.678290 21746 ts_tablet_manager.cc:1397] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.012s
I20260620 15:00:24.681553 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.664284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:39789)
0620 15:00:24.664856 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.664921 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.666042 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.667130 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.667139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.667595 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:24.668239 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.668259 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.670175 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.670181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.671668 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.671683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.672671 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.672678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.672929 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.673671 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:24.673698 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:24.675606 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:24.678338 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:24.678344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:24.678356 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:24.678677 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:24.679084 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:24.679087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:24.679089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:24.679241 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:24.679928 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:24.679935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:24.680195 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.680361 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.680655 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":147,"threads_started":1}
I20260620 15:00:24.681201 21746 raft_consensus.cc:348] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.681699 21746 raft_consensus.cc:729] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ded6fb27b242ab93da71151dffdacf, State: Initialized, Role: FOLLOWER
I20260620 15:00:24.682194 21746 consensus_queue.cc:260] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } } peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } }
I20260620 15:00:24.683177 21746 ts_tablet_manager.cc:1428] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 15:00:24.683401 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.664378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:39789 (local address 127.7.46.190:44819)
0620 15:00:24.664864 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.664883 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.665067 (+ 184us) server_negotiation.cc:407] Connection header received
0620 15:00:24.666449 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.666462 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.666741 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.666995 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:24.668424 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.670007 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.671849 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.672514 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.672692 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.675849 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:24.675855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:24.675870 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:24.675902 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:24.678198 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.678829 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.678834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.678839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.678976 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.679361 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.679364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.679366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.679800 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:24.682153 (+ 2353us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.682300 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.682603 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":143,"threads_started":1}
I20260620 15:00:24.685523 21854 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44819
I20260620 15:00:24.685755 21854 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:24.686256 21854 heartbeater.cc:511] Master 127.7.46.190:44819 requested a full tablet report, sending...
I20260620 15:00:24.688896 21666 ts_manager.cc:194] Registered new tserver with Master: d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405)
I20260620 15:00:24.692392 21666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:39789
I20260620 15:00:24.694909 21854 heartbeater.cc:503] Master 127.7.46.190:44819 was elected leader, sending a full tablet report...
I20260620 15:00:24.697847 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.691696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42405 (local address 127.0.0.1:35670)
0620 15:00:24.693230 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.693276 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.693410 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.694272 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.694278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.694302 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:24.694426 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.694432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.695491 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.695496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.696510 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.696523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.697286 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.697291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.697403 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.697447 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.697586 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.697711 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":147,"threads_started":1}
I20260620 15:00:24.697847 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.691946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.7.46.129:42405)
0620 15:00:24.692292 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.692303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.693533 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 15:00:24.693592 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.693595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.694102 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.694259 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:24.694541 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.695377 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.696642 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.697152 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.697284 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.697415 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.697619 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.697705 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":80,"threads_started":1}
I20260620 15:00:24.698736 21788 tablet_service.cc:1514] Processing DeleteTablet for tablet 178d4b9a152346439b3298f128092d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:24 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:35670
I20260620 15:00:24.699538 21863 tablet_replica.cc:330] stopping tablet replica
I20260620 15:00:24.699856 21863 raft_consensus.cc:2229] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:24.700098 21863 raft_consensus.cc:2258] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:24.701120 21863 ts_tablet_manager.cc:1905] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:24.705592 21863 ts_tablet_manager.cc:1918] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:24.705690 21863 log.cc:1196] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/178d4b9a152346439b3298f128092d9a
I20260620 15:00:24.705963 21863 ts_tablet_manager.cc:1939] T 178d4b9a152346439b3298f128092d9a P d5ded6fb27b242ab93da71151dffdacf: Deleting consensus metadata
I20260620 15:00:24.707816 21650 catalog_manager.cc:4841] TS d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405): tablet 178d4b9a152346439b3298f128092d9a (table test-table [id=34a27b28626345dcb5cb06181f20635f]) successfully deleted
W20260620 15:00:24.787070 21859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:24.787299 21859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:24.787339 21859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:24.791654 21859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:24.791711 21859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:24.791817 21859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:00:24.796947 21859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:33553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=35673
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:24.797828 21859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:24.798740 21859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:24.807798 21869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.807752 21868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.807797 21871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:24.807955 21859 server_base.cc:1034] running on GCE node
I20260620 15:00:24.808399 21859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:24.808985 21859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:24.810166 21859 hybrid_clock.cc:648] HybridClock initialized: now 1781967624810148 us; error 28 us; skew 500 ppm
Jun 20 15:00:24 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:24.819630 21859 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:00:24.820641 21859 webserver.cc:460] Webserver started at http://127.7.46.130:35673/ using document root <none> and password file <none>
I20260620 15:00:24.821259 21859 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:24.821369 21859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:24.825729 21859 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:24.827881 21878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.829180 21859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:24.829319 21859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "827c0252f33646deb8ceea442c97b632"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "93ab1c1f39bbe1eb6def67d7005c13f7"
server_key_iv: "9917db42a74401be9e0e622e0a629b34"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.829881 21859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:24.845798 21859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:24.848767 21859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:24.848968 21859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:24.849534 21859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:24.850340 21859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:24.850409 21859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.850484 21859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:24.850524 21859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.859408 21859 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:33553
I20260620 15:00:24.859438 21991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:33553 every 8 connection(s)
I20260620 15:00:24.860360 21859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:24.865113 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 21859
I20260620 15:00:24.865564 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:36067
--local_ip_for_outbound_sockets=127.7.46.131
--tserver_master_addrs=127.7.46.190:44819
--webserver_port=32973
--webserver_interface=127.7.46.131
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:00:24 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:24.874490 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.861982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:39593 (local address 127.7.46.190:44819)
0620 15:00:24.862143 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.862148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.862930 (+ 782us) server_negotiation.cc:407] Connection header received
0620 15:00:24.864131 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.864133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.864179 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.864244 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:24.865387 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.866205 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.867294 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.867700 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.867861 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.870519 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:24.870525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:24.870532 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:24.870567 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:24.872279 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.872753 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.872759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.872763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.872862 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.873233 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.873237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.873240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.873514 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:24.873629 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.874099 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.874250 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:00:24.875145 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.862372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:39593)
0620 15:00:24.862788 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.862829 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.863936 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.864409 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.864415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.864749 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:24.865241 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.865264 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.866321 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.866324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.867173 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.867182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.867908 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.867911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.868049 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.868605 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:24.868621 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:24.870334 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:24.872399 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:24.872403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:24.872416 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:24.872655 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:24.872987 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:24.872989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:24.872991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:24.873131 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:24.873630 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:24.873635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:24.873859 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.874018 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.874272 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260620 15:00:24.876495 21992 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44819
I20260620 15:00:24.876730 21992 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:24.877205 21992 heartbeater.cc:511] Master 127.7.46.190:44819 requested a full tablet report, sending...
I20260620 15:00:24.878232 21666 ts_manager.cc:194] Registered new tserver with Master: 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553)
I20260620 15:00:24.878803 21666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:39593
W20260620 15:00:24.976161 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:24.976455 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:24.976534 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:24.980985 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:24.981071 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:24.981163 21995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:00:24.986828 21995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:36067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=32973
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:24.988013 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:24.989218 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:24.998872 22003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.998872 22001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.998884 22000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:24.999114 21995 server_base.cc:1034] running on GCE node
I20260620 15:00:24.999575 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:25.000162 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:25.001335 21995 hybrid_clock.cc:648] HybridClock initialized: now 1781967625001302 us; error 45 us; skew 500 ppm
Jun 20 15:00:24 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:25.004022 21995 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:00:25.004853 21995 webserver.cc:460] Webserver started at http://127.7.46.131:32973/ using document root <none> and password file <none>
I20260620 15:00:25.005401 21995 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:25.005498 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:25.009132 21995 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:00:25.011104 22010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:25.012533 21995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:25.012681 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "290e882a16c04470a10d2de37109584c"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "2dc8176fa06f741f71c82e0178910a1c"
server_key_iv: "745e76f069daf18fa8111f43ac6e8f43"
server_key_version: "encryptionkey@0"
I20260620 15:00:25.013247 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:25.029790 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:25.032454 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:25.032641 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:25.033223 21995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:25.034052 21995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:25.034116 21995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:25.034183 21995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:25.034228 21995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:25.042896 21995 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:36067
I20260620 15:00:25.042919 22123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:36067 every 8 connection(s)
I20260620 15:00:25.043869 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:25.052337 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 21995
Jun 20 15:00:25 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:25.057842 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.045624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:57583 (local address 127.7.46.190:44819)
0620 15:00:25.045786 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.045790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.046599 (+ 809us) server_negotiation.cc:407] Connection header received
0620 15:00:25.047955 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.047959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.048021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.048131 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:25.049284 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.049812 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.050988 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.051220 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.051346 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.054039 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:25.054044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:25.054046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:25.054072 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:25.055628 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.056192 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.056197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.056200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.056311 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.056667 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.056670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.056672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.056855 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:25.056964 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.057473 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.057661 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":23680}
I20260620 15:00:25.058569 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.045993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:57583)
0620 15:00:25.046441 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:25.046483 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:25.047676 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:25.048310 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:25.048318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:25.048668 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:25.049127 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.049140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.049942 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.049946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.050855 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.050865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.051378 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.051383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.051560 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.052114 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:25.052131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:25.053883 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:25.055795 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:25.055803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:25.055816 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:25.056078 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:25.056414 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:25.056417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:25.056419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:25.056559 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:25.056978 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:25.056986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:25.057209 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 15:00:25.057395 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 15:00:25.057713 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
I20260620 15:00:25.059793 22124 heartbeater.cc:347] Connected to a master server at 127.7.46.190:44819
I20260620 15:00:25.060032 22124 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:25.060623 22124 heartbeater.cc:511] Master 127.7.46.190:44819 requested a full tablet report, sending...
I20260620 15:00:25.060930 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.053026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.7.46.190:44819)
0620 15:00:25.053292 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.053296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.053337 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:00:25.053514 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.053518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.053567 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.053672 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:25.054403 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.054912 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.055799 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.056039 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.056155 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.057412 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:25.057416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:25.057418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:25.057467 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:25.059211 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.059651 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.059654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.059655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.059806 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.060094 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.060098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.060099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.060315 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:25.060528 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.060633 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.060765 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"spinlock_wait_cycles":768,"thread_start_us":97,"threads_started":1}
I20260620 15:00:25.061655 21666 ts_manager.cc:194] Registered new tserver with Master: 290e882a16c04470a10d2de37109584c (127.7.46.131:36067)
I20260620 15:00:25.062242 21666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:57583
I20260620 15:00:25.062525 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:25.066608 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.063105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.7.46.190:44819)
0620 15:00:25.063206 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.063210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.063403 (+ 193us) server_negotiation.cc:407] Connection header received
0620 15:00:25.063579 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.063582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.063668 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.063758 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:25.064000 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.064485 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.065303 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.065531 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.065630 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.065930 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:25.066251 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:25.066385 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.066454 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.066503 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:00:25.067209 21666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:25.070277 21666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:25.077417 21788 tablet_service.cc:1467] Processing CreateTablet for tablet ef70504d38ba465c8f53811910cbe824 (DEFAULT_TABLE table=test-table [id=d326d6e9abcc4227839924809297e71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:25.077975 21788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef70504d38ba465c8f53811910cbe824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:25.080351 21746 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Bootstrap starting.
I20260620 15:00:25.080997 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.076432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:36067 (local address 127.0.0.1:54210)
0620 15:00:25.076524 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:25.076541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:25.076609 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:25.077479 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:25.077484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:25.077501 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:25.077593 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.077601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.078715 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.078718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.079731 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.079739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.080659 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.080663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.080752 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.080777 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:00:25.080827 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 15:00:25.080890 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 15:00:25.081230 21746 tablet_bootstrap.cc:654] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:25.081786 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.076582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.7.46.131:36067)
0620 15:00:25.076963 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.076976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.076997 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:00:25.077123 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.077127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.077325 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.077723 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:25.077822 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.078594 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.079875 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.080524 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.081173 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.081341 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.081392 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.081639 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":157,"threads_started":1}
I20260620 15:00:25.082257 21746 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: No bootstrap required, opened a new log
I20260620 15:00:25.082346 21746 ts_tablet_manager.cc:1397] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 15:00:25.083017 21746 raft_consensus.cc:348] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.083137 21746 raft_consensus.cc:374] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:25.083184 21746 raft_consensus.cc:729] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ded6fb27b242ab93da71151dffdacf, State: Initialized, Role: FOLLOWER
I20260620 15:00:25.083302 21746 consensus_queue.cc:260] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.083557 21746 ts_tablet_manager.cc:1428] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 15:00:25.084805 22058 tablet_service.cc:1467] Processing CreateTablet for tablet ef70504d38ba465c8f53811910cbe824 (DEFAULT_TABLE table=test-table [id=d326d6e9abcc4227839924809297e71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:25.085449 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.077103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:33553 (local address 127.0.0.1:43384)
0620 15:00:25.077355 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:25.077370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:25.077449 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:25.081221 (+ 3772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:25.081224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:25.081242 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:25.081328 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.081334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.082857 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.082860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.083985 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.083995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.085041 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.085045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.085139 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.085162 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:00:25.085245 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 15:00:25.085320 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260620 15:00:25.085507 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.080479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.7.46.130:33553)
0620 15:00:25.080835 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.080844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.080863 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:00:25.080923 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.080926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.081081 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.081231 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:25.081443 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.082689 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.084103 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.084919 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.085030 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.085165 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.085273 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.085360 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":168,"threads_started":1}
I20260620 15:00:25.085858 22058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef70504d38ba465c8f53811910cbe824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:25.090732 21926 tablet_service.cc:1467] Processing CreateTablet for tablet ef70504d38ba465c8f53811910cbe824 (DEFAULT_TABLE table=test-table [id=d326d6e9abcc4227839924809297e71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:25.091708 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef70504d38ba465c8f53811910cbe824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:25.093030 22137 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Bootstrap starting.
I20260620 15:00:25.095114 22137 tablet_bootstrap.cc:654] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:25.095844 22137 log.cc:824] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:25.098008 22137 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: No bootstrap required, opened a new log
I20260620 15:00:25.098178 22139 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Bootstrap starting.
I20260620 15:00:25.098193 22137 ts_tablet_manager.cc:1397] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:00:25.100234 22139 tablet_bootstrap.cc:654] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:25.100864 22139 log.cc:824] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:25.101099 22137 raft_consensus.cc:348] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.101320 22137 raft_consensus.cc:374] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:25.101392 22137 raft_consensus.cc:729] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290e882a16c04470a10d2de37109584c, State: Initialized, Role: FOLLOWER
I20260620 15:00:25.101805 22137 consensus_queue.cc:260] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.102478 22124 heartbeater.cc:503] Master 127.7.46.190:44819 was elected leader, sending a full tablet report...
I20260620 15:00:25.102655 22139 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: No bootstrap required, opened a new log
I20260620 15:00:25.102785 22139 ts_tablet_manager.cc:1397] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 15:00:25.102785 22137 ts_tablet_manager.cc:1428] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:25.105836 22139 raft_consensus.cc:348] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.106026 22139 raft_consensus.cc:374] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:25.106057 22139 raft_consensus.cc:729] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827c0252f33646deb8ceea442c97b632, State: Initialized, Role: FOLLOWER
I20260620 15:00:25.106359 22139 consensus_queue.cc:260] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.106864 21992 heartbeater.cc:503] Master 127.7.46.190:44819 was elected leader, sending a full tablet report...
I20260620 15:00:25.107110 22139 ts_tablet_manager.cc:1428] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260620 15:00:25.111176 21993 tablet.cc:2369] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:00:25.163319 21855 tablet.cc:2369] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:00:25.294800 22125 tablet.cc:2369] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:25.481667 22142 raft_consensus.cc:482] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:25.481902 22142 raft_consensus.cc:504] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.483018 22142 leader_election.cc:288] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290e882a16c04470a10d2de37109584c (127.7.46.131:36067), d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405)
I20260620 15:00:25.487083 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.483792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:42625 (local address 127.7.46.129:42405)
0620 15:00:25.484070 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.484075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.484091 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:00:25.484144 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.484147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.484190 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.484274 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:25.484536 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.485051 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.486096 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.486514 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.486638 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.486693 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.486866 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.486941 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260620 15:00:25.487083 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.483632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42405 (local address 127.7.46.130:42625)
0620 15:00:25.483885 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:25.483896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:25.483980 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:25.484287 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:25.484289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:25.484301 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:25.484431 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.484437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.485153 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.485156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.485981 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.485989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.486638 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.486642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.486742 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.486761 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 15:00:25.486838 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 15:00:25.486909 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260620 15:00:25.487087 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.483384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:36579 (local address 127.7.46.131:36067)
0620 15:00:25.483533 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.483540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.483636 (+ 96us) server_negotiation.cc:407] Connection header received
0620 15:00:25.483876 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.483881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.483942 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.483996 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:25.484420 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.484960 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.486124 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.486558 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.486680 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.486730 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.486903 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.486966 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:00:25.487108 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.483232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:36067 (local address 127.7.46.130:36579)
0620 15:00:25.483533 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:25.483550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:25.483697 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:25.484146 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:25.484148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:25.484166 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:25.484300 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.484307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.485069 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.485074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.485984 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.485993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.486672 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.486675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.486784 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.486807 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:00:25.486889 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 15:00:25.486959 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":44,"thread_start_us":104,"threads_started":1}
I20260620 15:00:25.487813 21808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70504d38ba465c8f53811910cbe824" candidate_uuid: "827c0252f33646deb8ceea442c97b632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ded6fb27b242ab93da71151dffdacf" is_pre_election: true
I20260620 15:00:25.487851 22078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70504d38ba465c8f53811910cbe824" candidate_uuid: "827c0252f33646deb8ceea442c97b632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290e882a16c04470a10d2de37109584c" is_pre_election: true
I20260620 15:00:25.488096 21808 raft_consensus.cc:2454] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827c0252f33646deb8ceea442c97b632 in term 0.
I20260620 15:00:25.488118 22078 raft_consensus.cc:2454] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827c0252f33646deb8ceea442c97b632 in term 0.
I20260620 15:00:25.488555 21879 leader_election.cc:302] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632; no voters:
I20260620 15:00:25.488786 22142 raft_consensus.cc:2781] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:25.488879 22142 raft_consensus.cc:482] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:25.488929 22142 raft_consensus.cc:3037] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:25.489912 22142 raft_consensus.cc:504] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.490219 22142 leader_election.cc:288] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [CANDIDATE]: Term 1 election: Requested vote from peers 290e882a16c04470a10d2de37109584c (127.7.46.131:36067), d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405)
I20260620 15:00:25.490568 21808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70504d38ba465c8f53811910cbe824" candidate_uuid: "827c0252f33646deb8ceea442c97b632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ded6fb27b242ab93da71151dffdacf"
I20260620 15:00:25.490589 22078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70504d38ba465c8f53811910cbe824" candidate_uuid: "827c0252f33646deb8ceea442c97b632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290e882a16c04470a10d2de37109584c"
I20260620 15:00:25.490696 22078 raft_consensus.cc:3037] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:25.490696 21808 raft_consensus.cc:3037] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:25.491905 22078 raft_consensus.cc:2454] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827c0252f33646deb8ceea442c97b632 in term 1.
I20260620 15:00:25.491905 21808 raft_consensus.cc:2454] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827c0252f33646deb8ceea442c97b632 in term 1.
I20260620 15:00:25.492175 21879 leader_election.cc:302] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632; no voters:
I20260620 15:00:25.492343 22142 raft_consensus.cc:2781] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:25.492545 22142 raft_consensus.cc:686] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 1 LEADER]: Becoming Leader. State: Replica: 827c0252f33646deb8ceea442c97b632, State: Running, Role: LEADER
I20260620 15:00:25.492879 22142 consensus_queue.cc:237] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.495849 21666 catalog_manager.cc:5495] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 reported cstate change: term changed from 0 to 1, leader changed from <none> to 827c0252f33646deb8ceea442c97b632 (127.7.46.130). New cstate: current_term: 1 leader_uuid: "827c0252f33646deb8ceea442c97b632" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:25.537165 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.533671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.7.46.130:33553)
0620 15:00:25.533815 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.533822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.533853 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:00:25.533946 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.533948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.534001 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.534110 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:25.534343 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.534901 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.535708 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.535976 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.536083 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.536383 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:25.536780 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:25.536918 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.536961 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.537018 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:00:25.543195 22078 raft_consensus.cc:1261] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 1 FOLLOWER]: Refusing update from remote peer 827c0252f33646deb8ceea442c97b632: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:25.543289 21808 raft_consensus.cc:1261] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Refusing update from remote peer 827c0252f33646deb8ceea442c97b632: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:25.543893 22142 consensus_queue.cc:1035] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:25.544094 22146 consensus_queue.cc:1035] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:25.557117 21661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:25.557251 21661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:25.559360 21661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Sending DeleteTablet for 3 replicas of tablet ef70504d38ba465c8f53811910cbe824
I20260620 15:00:25.560070 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 21704
I20260620 15:00:25.560014 22058 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70504d38ba465c8f53811910cbe824 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:25 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:54210
I20260620 15:00:25.560674 22159 tablet_replica.cc:330] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: stopping tablet replica
I20260620 15:00:25.560662 21926 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70504d38ba465c8f53811910cbe824 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:25 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:43384
I20260620 15:00:25.560967 22159 raft_consensus.cc:2229] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:25.561257 22159 raft_consensus.cc:2258] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:25.561610 22160 tablet_replica.cc:330] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: stopping tablet replica
I20260620 15:00:25.561946 22160 raft_consensus.cc:2229] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:25.562314 22160 raft_consensus.cc:2258] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:25.562891 22159 ts_tablet_manager.cc:1905] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:25.563686 22160 ts_tablet_manager.cc:1905] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:25.565657 22159 ts_tablet_manager.cc:1918] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:25.565830 22159 log.cc:1196] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/ef70504d38ba465c8f53811910cbe824
I20260620 15:00:25.566205 22159 ts_tablet_manager.cc:1939] T ef70504d38ba465c8f53811910cbe824 P 290e882a16c04470a10d2de37109584c: Deleting consensus metadata
I20260620 15:00:25.566993 22160 ts_tablet_manager.cc:1918] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:25.567148 22160 log.cc:1196] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/ef70504d38ba465c8f53811910cbe824
I20260620 15:00:25.567394 21650 catalog_manager.cc:4841] TS 290e882a16c04470a10d2de37109584c (127.7.46.131:36067): tablet ef70504d38ba465c8f53811910cbe824 (table test-table [id=d326d6e9abcc4227839924809297e71d]) successfully deleted
I20260620 15:00:25.567665 22160 ts_tablet_manager.cc:1939] T ef70504d38ba465c8f53811910cbe824 P 827c0252f33646deb8ceea442c97b632: Deleting consensus metadata
I20260620 15:00:25.568775 21653 catalog_manager.cc:4841] TS 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553): tablet ef70504d38ba465c8f53811910cbe824 (table test-table [id=d326d6e9abcc4227839924809297e71d]) successfully deleted
W20260620 15:00:25.569515 21650 connection.cc:535] client connection to 127.7.46.129:42405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:00:25.569598 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 21859
W20260620 15:00:25.569631 21650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:00:25.571750 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.571502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:42405 (local address 127.0.0.1:35672)
0620 15:00:25.571622 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:25.571685 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 15:00:25.571915 21650 catalog_manager.cc:4584] TS d5ded6fb27b242ab93da71151dffdacf (127.7.46.129:42405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef70504d38ba465c8f53811910cbe824: Network error: Client connection negotiation failed: client connection to 127.7.46.129:42405: connect: Connection refused (error 111)
I20260620 15:00:25.575662 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 21995
I20260620 15:00:25.580971 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 21633
I20260620 15:00:25.586897 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44819
--webserver_interface=127.7.46.190
--webserver_port=45039
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:44819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:25.690933 22162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:25.691154 22162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:25.691190 22162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:25.695181 22162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:25.695227 22162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:25.695246 22162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:25.695262 22162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:25.695277 22162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:25.700160 22162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:44819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=45039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:25.700986 22162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:25.701860 22162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:25.709990 22170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:25.710157 22162 server_base.cc:1034] running on GCE node
W20260620 15:00:25.709940 22167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:25.709944 22168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:25.710803 22162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:25.711860 22162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:25.713033 22162 hybrid_clock.cc:648] HybridClock initialized: now 1781967625713021 us; error 29 us; skew 500 ppm
Jun 20 15:00:25 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:25.715628 22162 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:00:25.716490 22162 webserver.cc:460] Webserver started at http://127.7.46.190:45039/ using document root <none> and password file <none>
I20260620 15:00:25.717003 22162 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:25.717087 22162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:25.720511 22162 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 15:00:25.722452 22177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:25.723567 22162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 15:00:25.723691 22162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "74111f51ba5643fdb56dccd2e93be8a5"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "6763372c23b88710db39814da18863a1"
server_key_iv: "541424056b03f9221fc71ed5b7a3f172"
server_key_version: "encryptionkey@0"
I20260620 15:00:25.724237 22162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:25.746871 22162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:25.750015 22162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:25.750205 22162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:25.757064 22162 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:44819
I20260620 15:00:25.757333 22229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:44819 every 8 connection(s)
I20260620 15:00:25.758076 22162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:25.762092 22230 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:00:25.762425 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 22162
I20260620 15:00:25.762852 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:42405
--local_ip_for_outbound_sockets=127.7.46.129
--tserver_master_addrs=127.7.46.190:44819
--webserver_port=44001
--webserver_interface=127.7.46.129
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:25.763811 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Bootstrap starting.
I20260620 15:00:25.780757 22230 log.cc:824] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:25.791316 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:00:25.791661 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5: Bootstrap complete.
I20260620 15:00:25.795703 22230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:25.796165 22230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74111f51ba5643fdb56dccd2e93be8a5, State: Initialized, Role: FOLLOWER
I20260620 15:00:25.796620 22230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:25.796761 22230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:25.796836 22230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:25.796955 22230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:00:25.798686 22230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:25.799000 22230 leader_election.cc:302] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74111f51ba5643fdb56dccd2e93be8a5; no voters:
I20260620 15:00:25.799278 22230 leader_election.cc:288] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:00:25.799403 22234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:00:25.799631 22234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [term 3 LEADER]: Becoming Leader. State: Replica: 74111f51ba5643fdb56dccd2e93be8a5, State: Running, Role: LEADER
I20260620 15:00:25.799993 22234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } }
I20260620 15:00:25.800405 22230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:25.801306 22235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "74111f51ba5643fdb56dccd2e93be8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } } }
I20260620 15:00:25.801424 22235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:25.802045 22243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:25.802685 22236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74111f51ba5643fdb56dccd2e93be8a5. Latest consensus state: current_term: 3 leader_uuid: "74111f51ba5643fdb56dccd2e93be8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74111f51ba5643fdb56dccd2e93be8a5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 44819 } } }
I20260620 15:00:25.802783 22236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74111f51ba5643fdb56dccd2e93be8a5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:25.807154 22243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:25.807824 22243 catalog_manager.cc:1259] Loaded cluster ID: 6f35081b35544658aa4bdb2e226a1808
I20260620 15:00:25.807880 22243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 15:00:25.808808 22243 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f96fe75c9ad google::LogMessage::Fail() at ??:0
@ 0x7f96fe75f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f96fe75c4de google::LogMessage::Flush() at ??:0
@ 0x7f96fe75da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f970909bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f970909cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9709099579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f97090dd640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f97009ae3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f97009aec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f97009b03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f970099f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9701fe36db start_thread at ??:0
@ 0x7f96fd25a61f clone at ??:0
W20260620 15:00:25.888417 22232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:25.888650 22232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:25.888746 22232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:25.892786 22232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:25.892858 22232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:25.892976 22232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:00:25.898172 22232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:42405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=44001
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:25.899133 22232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:25.900126 22232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:25.909229 22272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:25.909296 22271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:25.909646 22232 server_base.cc:1034] running on GCE node
W20260620 15:00:25.909658 22274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:25.910033 22232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:25.910642 22232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:25.911849 22232 hybrid_clock.cc:648] HybridClock initialized: now 1781967625911819 us; error 41 us; skew 500 ppm
Jun 20 15:00:25 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:25.914548 22232 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:00:25.915374 22232 webserver.cc:460] Webserver started at http://127.7.46.129:44001/ using document root <none> and password file <none>
I20260620 15:00:25.915967 22232 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:25.916052 22232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:25.919687 22232 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:00:25.921836 22281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:25.923091 22232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:25.923228 22232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "d5ded6fb27b242ab93da71151dffdacf"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "c444a8a9692e2b57856fe6f928e5ce2c"
server_key_iv: "461cc7702707f030fc4c2d5316a4be60"
server_key_version: "encryptionkey@0"
I20260620 15:00:25.923730 22232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:25.936707 22232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:25.940522 22232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:25.940744 22232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:25.941347 22232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:25.942497 22288 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 15:00:25.947821 22232 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 15:00:25.947911 22232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 15:00:25.948001 22232 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 15:00:25.949694 22232 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 15:00:25.949846 22232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 15:00:25.950021 22288 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Bootstrap starting.
I20260620 15:00:25.960711 22232 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:42405
I20260620 15:00:25.960794 22395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:42405 every 8 connection(s)
I20260620 15:00:25.961627 22232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:25.969462 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 22232
I20260620 15:00:25.969877 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:33553
--local_ip_for_outbound_sockets=127.7.46.130
--tserver_master_addrs=127.7.46.190:44819
--webserver_port=35673
--webserver_interface=127.7.46.130
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:25.971747 22288 log.cc:824] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:25.981298 22288 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:00:25.981714 22288 tablet_bootstrap.cc:492] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Bootstrap complete.
I20260620 15:00:25.983146 22288 ts_tablet_manager.cc:1397] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.002s
I20260620 15:00:25.986344 22288 raft_consensus.cc:348] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.986970 22288 raft_consensus.cc:729] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ded6fb27b242ab93da71151dffdacf, State: Initialized, Role: FOLLOWER
I20260620 15:00:25.987476 22288 consensus_queue.cc:260] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:25.988620 22288 ts_tablet_manager.cc:1428] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf: Time spent starting tablet: real 0.005s user 0.004s sys 0.004s
W20260620 15:00:26.065712 22398 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:25.963435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:57095)
0620 15:00:25.963913 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:25.963964 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:25.964729 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:26.064887 (+100158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
W20260620 15:00:26.066238 22396 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:44819 (0 consecutive failures): Network error: Failed to ping master at 127.7.46.190:44819: Client connection negotiation failed: client connection to 127.7.46.190:44819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:00:26.066927 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:26.066695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:55437)
0620 15:00:26.066812 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:26.066874 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 15:00:26.067587 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:26.067412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:40865)
0620 15:00:26.067503 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:26.067542 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 15:00:26.067842 22396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:00:26.083199 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:26.083413 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:26.083479 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:26.087405 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:26.087453 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:26.087544 22399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:00:26.092643 22399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:33553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=35673
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:26.093550 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:26.094547 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:26.103399 22409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:26.103377 22407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:26.103679 22399 server_base.cc:1034] running on GCE node
W20260620 15:00:26.103401 22406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:26.104086 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:26.104638 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:26.105775 22399 hybrid_clock.cc:648] HybridClock initialized: now 1781967626105760 us; error 28 us; skew 500 ppm
Jun 20 15:00:26 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:26.108266 22399 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:00:26.109099 22399 webserver.cc:460] Webserver started at http://127.7.46.130:35673/ using document root <none> and password file <none>
I20260620 15:00:26.109607 22399 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:26.109665 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:26.113099 22399 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:00:26.115032 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:26.116317 22399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:26.116437 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "827c0252f33646deb8ceea442c97b632"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "93ab1c1f39bbe1eb6def67d7005c13f7"
server_key_iv: "9917db42a74401be9e0e622e0a629b34"
server_key_version: "encryptionkey@0"
I20260620 15:00:26.116925 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:26.138378 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:26.141403 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:26.141593 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:26.142144 22399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:26.142946 22399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:26.142992 22399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:26.143029 22399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:26.143049 22399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:26.151670 22399 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:33553
I20260620 15:00:26.151731 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:33553 every 8 connection(s)
I20260620 15:00:26.152630 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:26.156272 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:26.154584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:48723)
0620 15:00:26.155022 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:26.155531 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":134,"threads_started":1}
I20260620 15:00:26.156418 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 22399
W20260620 15:00:26.156713 22530 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:44819 (0 consecutive failures): Network error: Failed to ping master at 127.7.46.190:44819: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
I20260620 15:00:26.156888 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:36067
--local_ip_for_outbound_sockets=127.7.46.131
--tserver_master_addrs=127.7.46.190:44819
--webserver_port=32973
--webserver_interface=127.7.46.131
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:26.157291 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:26.157105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:58731)
0620 15:00:26.157191 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:26.157242 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 15:00:26.157917 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:26.157736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:39237)
0620 15:00:26.157830 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:26.157869 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 15:00:26.158105 22530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:00:26.263751 22533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:26.263993 22533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:26.264032 22533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:26.267988 22533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:26.268035 22533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:26.268090 22533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:00:26.273090 22533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:36067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=32973
--tserver_master_addrs=127.7.46.190:44819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:26.273973 22533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:26.274876 22533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:26.284106 22541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:26.284118 22539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:26.284113 22538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:26.284220 22533 server_base.cc:1034] running on GCE node
I20260620 15:00:26.284787 22533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:26.285375 22533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:26.286537 22533 hybrid_clock.cc:648] HybridClock initialized: now 1781967626286509 us; error 47 us; skew 500 ppm
Jun 20 15:00:26 dist-test-slave-30t4 krb5kdc[21085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967626, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:26.289232 22533 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:00:26.290042 22533 webserver.cc:460] Webserver started at http://127.7.46.131:32973/ using document root <none> and password file <none>
I20260620 15:00:26.290572 22533 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:26.290632 22533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:26.294180 22533 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:26.296164 22548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:26.297235 22533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:26.297361 22533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "290e882a16c04470a10d2de37109584c"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-30t4"
server_key: "2dc8176fa06f741f71c82e0178910a1c"
server_key_iv: "745e76f069daf18fa8111f43ac6e8f43"
server_key_version: "encryptionkey@0"
I20260620 15:00:26.297847 22533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:26.318020 22533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:26.321166 22533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:26.321353 22533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:26.321887 22533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:26.322698 22533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:26.322765 22533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:26.322830 22533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:26.322861 22533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:26.331619 22533 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:36067
I20260620 15:00:26.331650 22661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:36067 every 8 connection(s)
I20260620 15:00:26.332580 22533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:26.332970 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 22533
I20260620 15:00:26.336412 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:26.334743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:45663)
0620 15:00:26.335564 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:26.335670 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":502,"threads_started":1}
W20260620 15:00:26.336818 22662 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:44819 (0 consecutive failures): Network error: Failed to ping master at 127.7.46.190:44819: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
I20260620 15:00:26.337330 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:26.337142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:39469)
0620 15:00:26.337237 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:26.337282 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 15:00:26.337846 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:26.337680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:51827)
0620 15:00:26.337764 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:26.337802 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 15:00:26.338019 22662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:00:27.068758 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:27.068351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:60445)
0620 15:00:27.068573 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:27.068623 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260620 15:00:27.159085 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:27.158677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:41055)
0620 15:00:27.158929 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:27.158984 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":66,"threads_started":1}
I20260620 15:00:27.338879 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:27.338502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:33089)
0620 15:00:27.338738 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:27.338789 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
I20260620 15:00:27.451612 22669 raft_consensus.cc:482] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:27.451864 22669 raft_consensus.cc:504] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:27.452947 22669 leader_election.cc:288] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 290e882a16c04470a10d2de37109584c (127.7.46.131:36067), 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553)
Jun 20 15:00:27 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.130@KRBTEST.COM
Jun 20 15:00:27 dist-test-slave-30t4 krb5kdc[21085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.131@KRBTEST.COM
I20260620 15:00:27.466372 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:27.453143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:33553 (local address 127.7.46.129:55115)
0620 15:00:27.453737 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:27.453751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:27.453868 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:27.455213 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:27.455218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:27.455732 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:27.456212 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:27.456226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.457453 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.457455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:27.458372 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:27.458381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.459087 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.459091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:27.459258 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:27.459974 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:27.459986 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:27.461440 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:27.463995 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:27.464002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:27.464019 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:27.464376 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:27.464722 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:27.464727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:27.464729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:27.464857 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:27.465434 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:27.465437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:27.465675 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 15:00:27.465854 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 15:00:27.466177 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":408,"spinlock_wait_cycles":30208,"thread_start_us":75,"threads_started":1}
I20260620 15:00:27.466384 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:27.453313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:55115 (local address 127.7.46.130:33553)
0620 15:00:27.453635 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:27.453663 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:27.453857 (+ 194us) server_negotiation.cc:407] Connection header received
0620 15:00:27.454530 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:27.454536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:27.454870 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:27.455163 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:27.456392 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.457304 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.458482 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.458950 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.459116 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:27.461662 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:27.461669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:27.461685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:27.461715 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:27.463857 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:27.464481 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:27.464489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:27.464498 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:27.464630 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:27.464958 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:27.464962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:27.464964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:27.465318 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:27.465463 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:27.465961 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 15:00:27.466205 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
I20260620 15:00:27.466408 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:27.453313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:42499 (local address 127.7.46.131:36067)
0620 15:00:27.453713 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:27.453735 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:27.453781 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:00:27.454554 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:27.454560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:27.454870 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:27.455163 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:27.456376 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.457308 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.458511 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.458942 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.459098 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:27.462120 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:27.462126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:27.462139 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:27.462173 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:27.464083 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:27.464541 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:27.464546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:27.464551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:27.464684 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:27.464946 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:27.464950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:27.464952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:27.465319 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:27.465453 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:27.465959 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 15:00:27.466218 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":84,"threads_started":1}
I20260620 15:00:27.466411 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:27.453143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:36067 (local address 127.7.46.129:42499)
0620 15:00:27.453255 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:27.453270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:27.453369 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:27.455213 (+ 1844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:27.455219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:27.455703 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:27.456211 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:27.456226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.457457 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.457460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:27.458374 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:27.458385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.459062 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.459066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:27.459262 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:27.459939 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:27.459955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:27.461898 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:27.464178 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:27.464181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:27.464182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:27.464409 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:27.464770 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:27.464774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:27.464776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:27.464858 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:27.465428 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:27.465433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:27.465675 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 15:00:27.465854 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 15:00:27.466255 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":292,"spinlock_wait_cycles":21120}
W20260620 15:00:27.469300 22285 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
W20260620 15:00:27.469970 22282 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 290e882a16c04470a10d2de37109584c (127.7.46.131:36067): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
I20260620 15:00:27.470037 22282 leader_election.cc:302] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5ded6fb27b242ab93da71151dffdacf; no voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632
I20260620 15:00:27.470296 22669 raft_consensus.cc:2726] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:00:28.069892 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:28.069472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:55679)
0620 15:00:28.069740 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:28.069794 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260620 15:00:28.160341 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:28.159964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:52355)
0620 15:00:28.160207 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:28.160257 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260620 15:00:28.339984 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:28.339521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:33323)
0620 15:00:28.339813 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:28.339876 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
I20260620 15:00:29.071017 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.070607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:45047)
0620 15:00:29.070868 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.070922 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":69,"threads_started":1}
I20260620 15:00:29.132876 22677 raft_consensus.cc:482] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:29.132993 22677 raft_consensus.cc:504] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:29.133461 22677 leader_election.cc:288] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 290e882a16c04470a10d2de37109584c (127.7.46.131:36067), 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553)
W20260620 15:00:29.134218 22285 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
W20260620 15:00:29.134301 22282 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 290e882a16c04470a10d2de37109584c (127.7.46.131:36067): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
I20260620 15:00:29.134409 22282 leader_election.cc:302] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5ded6fb27b242ab93da71151dffdacf; no voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632
I20260620 15:00:29.134577 22677 raft_consensus.cc:2726] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:00:29.161324 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.160952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:48511)
0620 15:00:29.161172 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.161223 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
I20260620 15:00:29.340991 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.340625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:60115)
0620 15:00:29.340852 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.340908 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260620 15:00:30.072180 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:30.071740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:33891)
0620 15:00:30.072040 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:30.072088 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":79,"threads_started":1}
I20260620 15:00:30.162482 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:30.162009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:52191)
0620 15:00:30.162281 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:30.162378 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260620 15:00:30.342038 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:30.341663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:55075)
0620 15:00:30.341898 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:30.341950 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260620 15:00:31.073486 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.073048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:53131)
0620 15:00:31.073327 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:31.073379 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
I20260620 15:00:31.163583 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.163207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:39221)
0620 15:00:31.163443 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:31.163496 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260620 15:00:31.343078 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.342731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:42913)
0620 15:00:31.342944 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:31.342995 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260620 15:00:32.074671 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.074216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:43323)
0620 15:00:32.074516 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:32.074571 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
I20260620 15:00:32.164633 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.164255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:45791)
0620 15:00:32.164490 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:32.164544 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
I20260620 15:00:32.172331 22688 raft_consensus.cc:482] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:32.172439 22688 raft_consensus.cc:504] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:32.172850 22688 leader_election.cc:288] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 290e882a16c04470a10d2de37109584c (127.7.46.131:36067), 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553)
W20260620 15:00:32.173539 22285 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
W20260620 15:00:32.173640 22282 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 290e882a16c04470a10d2de37109584c (127.7.46.131:36067): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
I20260620 15:00:32.173739 22282 leader_election.cc:302] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5ded6fb27b242ab93da71151dffdacf; no voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632
I20260620 15:00:32.173905 22688 raft_consensus.cc:2726] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:00:32.344156 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.343753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:46899)
0620 15:00:32.344020 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:32.344070 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":68,"threads_started":1}
I20260620 15:00:33.075946 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:33.075440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:60777)
0620 15:00:33.075710 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:33.075846 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260620 15:00:33.165663 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:33.165257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:36045)
0620 15:00:33.165531 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:33.165582 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260620 15:00:33.345368 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:33.344870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:54195)
0620 15:00:33.345146 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:33.345250 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":66,"threads_started":1}
I20260620 15:00:34.077049 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:34.076666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:36045)
0620 15:00:34.076897 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:34.076948 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260620 15:00:34.166668 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:34.166296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:34187)
0620 15:00:34.166536 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:34.166585 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260620 15:00:34.346390 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:34.346027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:35505)
0620 15:00:34.346250 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:34.346301 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260620 15:00:35.078321 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:35.077881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:57901)
0620 15:00:35.078128 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:35.078189 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20260620 15:00:35.167726 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:35.167367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:35005)
0620 15:00:35.167586 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:35.167636 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260620 15:00:35.347434 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:35.347060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:43989)
0620 15:00:35.347291 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:35.347344 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260620 15:00:35.570576 22699 raft_consensus.cc:482] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:35.570732 22699 raft_consensus.cc:504] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:35.571228 22699 leader_election.cc:288] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 290e882a16c04470a10d2de37109584c (127.7.46.131:36067), 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553)
W20260620 15:00:35.572052 22282 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 290e882a16c04470a10d2de37109584c (127.7.46.131:36067): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
W20260620 15:00:35.572197 22285 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
I20260620 15:00:35.572314 22285 leader_election.cc:302] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5ded6fb27b242ab93da71151dffdacf; no voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632
I20260620 15:00:35.572501 22699 raft_consensus.cc:2726] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:00:36.079553 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:36.079178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:35715)
0620 15:00:36.079410 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:36.079459 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20260620 15:00:36.168882 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:36.168495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:41861)
0620 15:00:36.168740 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:36.168794 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":112,"threads_started":1}
I20260620 15:00:36.348577 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:36.348199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:39271)
0620 15:00:36.348432 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:36.348485 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260620 15:00:37.080742 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:37.080289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:60541)
0620 15:00:37.080568 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:37.080623 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260620 15:00:37.169976 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:37.169579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:57315)
0620 15:00:37.169834 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:37.169888 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260620 15:00:37.349694 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:37.349315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:60397)
0620 15:00:37.349552 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:37.349604 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260620 15:00:38.081844 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:38.081465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:41017)
0620 15:00:38.081703 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:38.081751 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260620 15:00:38.171185 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:38.170794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:34861)
0620 15:00:38.171034 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:38.171097 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
I20260620 15:00:38.350767 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:38.350396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:49905)
0620 15:00:38.350627 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:38.350678 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
I20260620 15:00:39.082911 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.082526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:33429)
0620 15:00:39.082760 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:39.082816 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260620 15:00:39.172350 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.171948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:56703)
0620 15:00:39.172196 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:39.172249 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260620 15:00:39.351816 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.351435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:45643)
0620 15:00:39.351650 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:39.351699 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260620 15:00:40.084100 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:40.083639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:35883)
0620 15:00:40.083941 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:40.083998 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":76,"threads_started":1}
I20260620 15:00:40.173373 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:40.172988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:40297)
0620 15:00:40.173221 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:40.173273 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260620 15:00:40.352931 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:40.352551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.131:39223)
0620 15:00:40.352781 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:40.352841 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260620 15:00:40.408411 22715 raft_consensus.cc:482] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:40.408529 22715 raft_consensus.cc:504] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e882a16c04470a10d2de37109584c" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 36067 } } peers { permanent_uuid: "827c0252f33646deb8ceea442c97b632" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33553 } } peers { permanent_uuid: "d5ded6fb27b242ab93da71151dffdacf" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 42405 } }
I20260620 15:00:40.408967 22715 leader_election.cc:288] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 290e882a16c04470a10d2de37109584c (127.7.46.131:36067), 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553)
W20260620 15:00:40.409793 22282 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 290e882a16c04470a10d2de37109584c (127.7.46.131:36067): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
W20260620 15:00:40.409904 22285 leader_election.cc:341] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 827c0252f33646deb8ceea442c97b632 (127.7.46.130:33553): Not found: Tablet not found: ef70504d38ba465c8f53811910cbe824
I20260620 15:00:40.409971 22285 leader_election.cc:302] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5ded6fb27b242ab93da71151dffdacf; no voters: 290e882a16c04470a10d2de37109584c, 827c0252f33646deb8ceea442c97b632
I20260620 15:00:40.410099 22715 raft_consensus.cc:2726] T ef70504d38ba465c8f53811910cbe824 P d5ded6fb27b242ab93da71151dffdacf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:00:41.085458 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:41.085079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.129:45343)
0620 15:00:41.085315 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:41.085365 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260620 15:00:41.174580 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:41.174149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:44819 (local address 127.7.46.130:40969)
0620 15:00:41.174435 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:41.174493 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260620 15:00:41.333918 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 22232
I20260620 15:00:41.341150 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 22399
I20260620 15:00:41.346603 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 22533
2026-06-20T15:00:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19412 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:41 dist-test-slave-30t4 krb5kdc[22720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:41 dist-test-slave-30t4 krb5kdc[22720](info): set up 2 sockets
Jun 20 15:00:41 dist-test-slave-30t4 krb5kdc[22720](info): commencing operation
krb5kdc: starting...
W20260620 15:00:42.392267 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:00:42 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:00:42.514490 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43303
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:42.619619 22733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:42.619928 22733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:42.620008 22733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:42.623901 22733 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:00:42.623975 22733 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:00:42.624006 22733 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:00:42.624039 22733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:42.624083 22733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:42.624125 22733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:42.624171 22733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:42.624205 22733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:42.629390 22733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:42.630376 22733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:42.631358 22733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:42.639866 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:42.639933 22733 server_base.cc:1034] running on GCE node
W20260620 15:00:42.639870 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:42.639871 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:42.640614 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:42.641642 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:42.642845 22733 hybrid_clock.cc:648] HybridClock initialized: now 1781967642642822 us; error 33 us; skew 500 ppm
Jun 20 15:00:42 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:42.645617 22733 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:00:42.646500 22733 webserver.cc:460] Webserver started at http://127.7.46.190:45585/ using document root <none> and password file <none>
I20260620 15:00:42.647053 22733 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:42.647140 22733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:42.647362 22733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:42.649355 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "500bb606d97d499a8d5b1a233de20350"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "ca4a3ae3f3c87ec6fe8dc06c4f5dee49"
server_key_iv: "5fdbc11161247bcc92cfaded7c7507ce"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.649904 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "500bb606d97d499a8d5b1a233de20350"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "ca4a3ae3f3c87ec6fe8dc06c4f5dee49"
server_key_iv: "5fdbc11161247bcc92cfaded7c7507ce"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.653398 22733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 15:00:42.655683 22748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:42.656970 22733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:42.657124 22733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "500bb606d97d499a8d5b1a233de20350"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "ca4a3ae3f3c87ec6fe8dc06c4f5dee49"
server_key_iv: "5fdbc11161247bcc92cfaded7c7507ce"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.657233 22733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:42.684691 22733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:42.687454 22733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:42.687655 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:42.694351 22733 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:43303
I20260620 15:00:42.694639 22800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:43303 every 8 connection(s)
I20260620 15:00:42.695262 22733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:42.697853 22801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:42.700103 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 22733
I20260620 15:00:42.700228 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:00:42.700527 7354 external_mini_cluster.cc:1428] Setting key e06010c9d9e254ecd4a7ea466577c463
I20260620 15:00:42.703753 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Bootstrap starting.
I20260620 15:00:42.706110 22801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:42.706875 22801 log.cc:824] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:42.708796 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: No bootstrap required, opened a new log
Jun 20 15:00:42 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:42.713475 22801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:00:42.713732 22801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:42.713790 22801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500bb606d97d499a8d5b1a233de20350, State: Initialized, Role: FOLLOWER
I20260620 15:00:42.714231 22801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:00:42.714387 22801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:42.714450 22801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:42.714537 22801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:42.714686 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:42.701804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.7.46.190:43303)
0620 15:00:42.702171 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:42.702192 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:42.702227 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:00:42.702857 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:42.702872 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:42.703178 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:42.703435 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:42.704314 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.705269 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.706419 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.706868 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.707020 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:42.709699 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:42.709706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:42.709721 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:42.709749 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:42.711810 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:42.712206 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:42.712213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:42.712219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:42.712356 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:42.712608 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:42.712612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:42.712615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:42.713098 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:42.713299 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:42.713488 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 15:00:42.713783 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260620 15:00:42.715541 22801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:00:42.716830 22801 leader_election.cc:302] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500bb606d97d499a8d5b1a233de20350; no voters:
I20260620 15:00:42.717096 22801 leader_election.cc:288] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:42.717213 22806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:42.717458 22806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 1 LEADER]: Becoming Leader. State: Replica: 500bb606d97d499a8d5b1a233de20350, State: Running, Role: LEADER
I20260620 15:00:42.717778 22806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:00:42.718286 22801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:42.719658 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500bb606d97d499a8d5b1a233de20350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } } }
I20260620 15:00:42.719697 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500bb606d97d499a8d5b1a233de20350. Latest consensus state: current_term: 1 leader_uuid: "500bb606d97d499a8d5b1a233de20350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } } }
I20260620 15:00:42.719810 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:42.719825 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:42.720163 22817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:42.723127 22817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:42.728231 22817 catalog_manager.cc:1347] Generated new cluster ID: 5716e8328a2a4a318a4f8d5ad34bec6d
I20260620 15:00:42.728324 22817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:42.734256 22817 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:42.735082 22817 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:42.740657 22817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Generated new TSK 0
I20260620 15:00:42.741248 22817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 15:00:42.776762 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43303
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:42.906185 22827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:42.906406 22827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:42.906455 22827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:42.910317 22827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:42.910365 22827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:42.910419 22827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:00:42.915424 22827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:42.916347 22827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:42.917235 22827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:42.926838 22832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:42.926965 22827 server_base.cc:1034] running on GCE node
W20260620 15:00:42.926846 22833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:42.926846 22835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:42.927626 22827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:42.928254 22827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:42.929455 22827 hybrid_clock.cc:648] HybridClock initialized: now 1781967642929442 us; error 30 us; skew 500 ppm
Jun 20 15:00:42 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:42.932204 22827 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:00:42.933050 22827 webserver.cc:460] Webserver started at http://127.7.46.129:35003/ using document root <none> and password file <none>
I20260620 15:00:42.933609 22827 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:42.933699 22827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:42.933893 22827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:42.935815 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "378e968466b446ff8221671810afd5eb"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "a321b9c445d26902f2fe37c382a85d71"
server_key_iv: "8f67c9e15c53ca1e31cfdb7a2fdaa74b"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.936381 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "378e968466b446ff8221671810afd5eb"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "a321b9c445d26902f2fe37c382a85d71"
server_key_iv: "8f67c9e15c53ca1e31cfdb7a2fdaa74b"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.939952 22827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:42.942220 22842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:42.943506 22827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:42.943648 22827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "378e968466b446ff8221671810afd5eb"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "a321b9c445d26902f2fe37c382a85d71"
server_key_iv: "8f67c9e15c53ca1e31cfdb7a2fdaa74b"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.943764 22827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:42.979676 22827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:42.982896 22827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:42.983093 22827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:42.983709 22827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:42.984551 22827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:42.984622 22827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:42.984688 22827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:42.984733 22827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:42.993194 22827 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:36425
I20260620 15:00:42.993221 22955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:36425 every 8 connection(s)
I20260620 15:00:42.994036 22827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
Jun 20 15:00:42 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:43.003233 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 22827
I20260620 15:00:43.003387 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:43.003677 7354 external_mini_cluster.cc:1428] Setting key 890b93ee6ff84328d8d41de9a882775b
I20260620 15:00:43.007949 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:42.995810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:37689 (local address 127.7.46.190:43303)
0620 15:00:42.995950 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:42.995955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:42.996788 (+ 833us) server_negotiation.cc:407] Connection header received
0620 15:00:42.998012 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:42.998015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:42.998061 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:42.998137 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:42.999196 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.999714 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.000973 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.001257 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.001362 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.003883 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:43.003887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:43.003890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:43.003917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:43.005554 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.006072 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.006079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.006084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.006182 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.006495 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.006501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.006506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.006761 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:43.006865 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.007592 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.007708 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:00:43.008812 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:42.996195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:37689)
0620 15:00:42.996639 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:42.996679 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:42.997797 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:42.998300 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:42.998307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:42.998627 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:42.999067 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:42.999079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.999971 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.999973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.000836 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.000845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.001435 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.001438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.001596 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.002104 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:43.002120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:43.003671 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:43.005683 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.005688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.005707 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.005955 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.006273 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.006275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.006276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.006397 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.006895 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:43.006903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:43.007229 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.007432 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.007791 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":101,"threads_started":1}
I20260620 15:00:43.010027 22956 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43303
I20260620 15:00:43.010267 22956 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:43.010723 22956 heartbeater.cc:511] Master 127.7.46.190:43303 requested a full tablet report, sending...
I20260620 15:00:43.012192 22765 ts_manager.cc:194] Registered new tserver with Master: 378e968466b446ff8221671810afd5eb (127.7.46.129:36425)
I20260620 15:00:43.013276 22765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:37689
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 15:00:43.031383 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43303
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:43.138324 22961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:43.138571 22961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:43.138612 22961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:43.142899 22961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:43.142961 22961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:43.143062 22961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:00:43.148445 22961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:43.149400 22961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:43.150364 22961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:00:43.160055 22961 server_base.cc:1034] running on GCE node
W20260620 15:00:43.160063 22966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:43.160208 22969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:43.160069 22967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:43.160705 22961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:43.161293 22961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:43.162453 22961 hybrid_clock.cc:648] HybridClock initialized: now 1781967643162407 us; error 56 us; skew 500 ppm
Jun 20 15:00:43 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967643, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:43.165119 22961 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:00:43.165891 22961 webserver.cc:460] Webserver started at http://127.7.46.130:40707/ using document root <none> and password file <none>
I20260620 15:00:43.166478 22961 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:43.166565 22961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:43.166791 22961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:43.168752 22961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "f3d61096013345c38e38c07bbeac36bc"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "a8db5ab978612184d79659ef2e059627"
server_key_iv: "c7da6e27c2f57a01ed5f0e19abc27d61"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.169296 22961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "f3d61096013345c38e38c07bbeac36bc"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "a8db5ab978612184d79659ef2e059627"
server_key_iv: "c7da6e27c2f57a01ed5f0e19abc27d61"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.172766 22961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:00:43.174969 22976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.176172 22961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:00:43.176302 22961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "f3d61096013345c38e38c07bbeac36bc"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "a8db5ab978612184d79659ef2e059627"
server_key_iv: "c7da6e27c2f57a01ed5f0e19abc27d61"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.176406 22961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:43.187129 22961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:43.189772 22961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:43.189952 22961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:43.190426 22961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:43.191098 22961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:43.191143 22961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.191179 22961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:43.191206 22961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.199414 22961 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:34213
I20260620 15:00:43.199453 23089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:34213 every 8 connection(s)
I20260620 15:00:43.200219 22961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:43.207118 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 22961
I20260620 15:00:43.207268 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:43.207633 7354 external_mini_cluster.cc:1428] Setting key 82f17093524b0baefdbc73c5042fbc0d
Jun 20 15:00:43 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967643, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:43.213768 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.201886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:56653 (local address 127.7.46.190:43303)
0620 15:00:43.202043 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.202046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.202744 (+ 698us) server_negotiation.cc:407] Connection header received
0620 15:00:43.203976 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.203979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.204025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.204098 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:43.205140 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.205649 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.206707 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.206945 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.207046 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.209785 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:43.209789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:43.209794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:43.209821 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:43.211483 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.212065 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.212071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.212076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.212167 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.212548 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.212555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.212559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.212810 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:43.212912 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.213393 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.213535 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:00:43.214529 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.202208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:56653)
0620 15:00:43.202608 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.202648 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.203741 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.204246 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.204252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.204569 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:43.205024 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.205036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.205761 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.205763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.206585 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.206593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.207136 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.207143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.207364 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.208002 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:43.208024 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:43.209602 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:43.211617 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.211621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.211632 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.211963 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.212303 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.212306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.212307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.212439 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.212910 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:43.212915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:43.213122 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.213264 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.213585 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20260620 15:00:43.215873 23090 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43303
I20260620 15:00:43.216106 23090 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:43.216564 23090 heartbeater.cc:511] Master 127.7.46.190:43303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
I20260620 15:00:43.217593 22765 ts_manager.cc:194] Registered new tserver with Master: f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213)
I20260620 15:00:43.218163 22765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:56653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 15:00:43.235247 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43303
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:43.340421 23095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:43.340636 23095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:43.340675 23095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:43.344573 23095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:43.344620 23095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:43.344674 23095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:00:43.349733 23095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:00:43.350574 23095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:43.351444 23095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:43.360597 23101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:43.360649 23095 server_base.cc:1034] running on GCE node
W20260620 15:00:43.360600 23100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:43.360934 23103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:43.361363 23095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:43.361953 23095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:43.363135 23095 hybrid_clock.cc:648] HybridClock initialized: now 1781967643363106 us; error 41 us; skew 500 ppm
Jun 20 15:00:43 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967643, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:43.365891 23095 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:00:43.366659 23095 webserver.cc:460] Webserver started at http://127.7.46.131:42929/ using document root <none> and password file <none>
I20260620 15:00:43.367183 23095 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:43.367271 23095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:43.367491 23095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:43.369413 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "9186149806774e538dfd30894999724f"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "c63efda3cc419b184a492bb5fb065fd6"
server_key_iv: "373369966de598371c39538de99e9809"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.370002 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "9186149806774e538dfd30894999724f"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "c63efda3cc419b184a492bb5fb065fd6"
server_key_iv: "373369966de598371c39538de99e9809"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.373504 23095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:00:43.375645 23110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.376807 23095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:00:43.376935 23095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "9186149806774e538dfd30894999724f"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "c63efda3cc419b184a492bb5fb065fd6"
server_key_iv: "373369966de598371c39538de99e9809"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.377038 23095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:43.385772 23095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:43.388788 23095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:43.388964 23095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:43.389469 23095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:43.390249 23095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:43.390326 23095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.390388 23095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:43.390439 23095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.398501 23095 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:42993
I20260620 15:00:43.398598 23223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:42993 every 8 connection(s)
I20260620 15:00:43.399328 23095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:43.400532 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 23095
I20260620 15:00:43.400671 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:43.401000 7354 external_mini_cluster.cc:1428] Setting key ec14d789e66bb1326063019fd12c75fc
Jun 20 15:00:43 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967643, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:00:43.411916 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.401121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:51333 (local address 127.7.46.190:43303)
0620 15:00:43.401263 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.401267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.402171 (+ 904us) server_negotiation.cc:407] Connection header received
0620 15:00:43.403578 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.403580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.403623 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.403690 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:43.404790 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.405307 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.406357 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.406569 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.406646 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.408997 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:43.409000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:43.409002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:43.409024 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:43.410245 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.410669 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.410672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.410673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.410747 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.411021 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.411023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.411024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.411179 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:43.411262 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.411645 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.411734 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:00:43.412619 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.401509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:51333)
0620 15:00:43.402017 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.402069 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.403399 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.403911 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.403917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.404238 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:43.404666 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.404678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.405404 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.405407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.406248 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.406257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.406687 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.406690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.406840 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.407317 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:43.407331 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:43.408853 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:43.410346 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.410350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.410360 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.410587 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.410824 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.410826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.410828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.410944 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.411263 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:43.411267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:43.411458 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.411599 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.411880 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260620 15:00:43.413684 23224 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43303
I20260620 15:00:43.413952 23224 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:43.414433 23224 heartbeater.cc:511] Master 127.7.46.190:43303 requested a full tablet report, sending...
I20260620 15:00:43.415397 22765 ts_manager.cc:194] Registered new tserver with Master: 9186149806774e538dfd30894999724f (127.7.46.131:42993)
I20260620 15:00:43.416023 22765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:51333
I20260620 15:00:43.426527 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:43.435089 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.428476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.7.46.190:43303)
0620 15:00:43.428592 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.428595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.428776 (+ 181us) server_negotiation.cc:407] Connection header received
0620 15:00:43.428858 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.428860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.428896 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.428962 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:43.429588 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.430066 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.430816 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.431015 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.431086 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.432463 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:43.432466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:43.432468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:43.432489 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:43.433692 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.434055 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.434058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.434059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.434135 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.434391 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.434394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.434395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.434609 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:43.434727 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.434888 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.434973 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:00:43.437758 22765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:43.440656 22765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:43.454465 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.449237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:42993 (local address 127.0.0.1:55590)
0620 15:00:43.449886 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.449896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.449952 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.450616 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.450618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.450635 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:43.450715 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.450722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.451767 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.451773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.452995 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.453005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.453920 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.453924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.454027 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.454071 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.454194 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.454309 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":137,"thread_start_us":93,"threads_started":1}
I20260620 15:00:43.454491 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.449374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.7.46.131:42993)
0620 15:00:43.450171 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.450183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.450205 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:00:43.450281 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.450286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.450474 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.450627 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:43.450845 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.451600 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.453116 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.453795 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.453897 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.454030 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.454237 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.454336 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":134,"threads_started":1}
I20260620 15:00:43.455096 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.448544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:34213 (local address 127.0.0.1:44512)
0620 15:00:43.448866 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.449623 (+ 757us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.449696 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.450226 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.450232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.450256 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:43.450373 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.450379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.451905 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.451908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.453442 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.453454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.454592 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.454596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.454750 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.454794 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.454869 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.454943 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":162,"thread_start_us":95,"threads_started":1}
I20260620 15:00:43.455132 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.448932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:36425 (local address 127.0.0.1:45150)
0620 15:00:43.449528 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.449541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.449674 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.450385 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.450388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.450412 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:43.450526 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.450536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.451908 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.451910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.453442 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.453454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.454622 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.454626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.454794 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.454813 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.454907 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.454980 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":213,"thread_start_us":107,"threads_started":1}
I20260620 15:00:43.455453 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.449374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.7.46.129:36425)
0620 15:00:43.449716 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.449731 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.449748 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:00:43.449846 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.449852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.450050 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.450212 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:43.450664 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.451759 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.453571 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.454324 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.454421 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.454551 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.455236 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.455328 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":81,"threads_started":1}
I20260620 15:00:43.456051 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.448682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.7.46.130:34213)
0620 15:00:43.449107 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.449119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.449692 (+ 573us) server_negotiation.cc:407] Connection header received
0620 15:00:43.449848 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.449852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.450050 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.450212 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:43.450490 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.451696 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.453596 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.454305 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.454397 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.454524 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.455829 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.455918 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":113,"threads_started":1}
I20260620 15:00:43.458032 23158 tablet_service.cc:1467] Processing CreateTablet for tablet 23feaacc9fb44fe8ae95521145494d5b (DEFAULT_TABLE table=test-table [id=257f2994fc4c4351b658dd3655cdee37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:43.458309 22890 tablet_service.cc:1467] Processing CreateTablet for tablet 23feaacc9fb44fe8ae95521145494d5b (DEFAULT_TABLE table=test-table [id=257f2994fc4c4351b658dd3655cdee37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:43.458312 23024 tablet_service.cc:1467] Processing CreateTablet for tablet 23feaacc9fb44fe8ae95521145494d5b (DEFAULT_TABLE table=test-table [id=257f2994fc4c4351b658dd3655cdee37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:43.459173 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23feaacc9fb44fe8ae95521145494d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:43.459179 22890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23feaacc9fb44fe8ae95521145494d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:43.459185 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23feaacc9fb44fe8ae95521145494d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:43.465515 23241 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Bootstrap starting.
I20260620 15:00:43.467226 23243 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Bootstrap starting.
I20260620 15:00:43.468037 23241 tablet_bootstrap.cc:654] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:43.468149 23242 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Bootstrap starting.
I20260620 15:00:43.468825 23241 log.cc:824] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:43.469895 23243 tablet_bootstrap.cc:654] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:43.470351 23242 tablet_bootstrap.cc:654] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:43.470696 23241 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: No bootstrap required, opened a new log
I20260620 15:00:43.470824 23243 log.cc:824] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:43.470907 23241 ts_tablet_manager.cc:1397] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 15:00:43.471634 23242 log.cc:824] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:43.472949 23243 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: No bootstrap required, opened a new log
I20260620 15:00:43.473140 23243 ts_tablet_manager.cc:1397] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 15:00:43.473640 23242 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: No bootstrap required, opened a new log
I20260620 15:00:43.473814 23242 ts_tablet_manager.cc:1397] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 15:00:43.474179 23241 raft_consensus.cc:348] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.474406 23241 raft_consensus.cc:374] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:43.474458 23241 raft_consensus.cc:729] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d61096013345c38e38c07bbeac36bc, State: Initialized, Role: FOLLOWER
I20260620 15:00:43.474879 23241 consensus_queue.cc:260] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.475463 23090 heartbeater.cc:503] Master 127.7.46.190:43303 was elected leader, sending a full tablet report...
I20260620 15:00:43.475837 23241 ts_tablet_manager.cc:1428] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:43.477288 23243 raft_consensus.cc:348] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.477558 23243 raft_consensus.cc:374] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:43.477634 23243 raft_consensus.cc:729] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9186149806774e538dfd30894999724f, State: Initialized, Role: FOLLOWER
I20260620 15:00:43.477850 23242 raft_consensus.cc:348] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.478096 23242 raft_consensus.cc:374] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:43.478066 23243 consensus_queue.cc:260] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.478150 23242 raft_consensus.cc:729] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378e968466b446ff8221671810afd5eb, State: Initialized, Role: FOLLOWER
I20260620 15:00:43.478569 23242 consensus_queue.cc:260] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.479094 23243 ts_tablet_manager.cc:1428] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 15:00:43.479439 23224 heartbeater.cc:503] Master 127.7.46.190:43303 was elected leader, sending a full tablet report...
I20260620 15:00:43.479871 22956 heartbeater.cc:503] Master 127.7.46.190:43303 was elected leader, sending a full tablet report...
I20260620 15:00:43.480576 23242 ts_tablet_manager.cc:1428] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260620 15:00:43.495204 22957 tablet.cc:2369] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:00:43.650451 23225 tablet.cc:2369] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:00:43.701224 23091 tablet.cc:2369] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:43.798985 23247 raft_consensus.cc:482] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:43.799211 23247 raft_consensus.cc:504] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.800455 23247 leader_election.cc:288] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9186149806774e538dfd30894999724f (127.7.46.131:42993), 378e968466b446ff8221671810afd5eb (127.7.46.129:36425)
I20260620 15:00:43.804472 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.800835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:40737 (local address 127.7.46.129:36425)
0620 15:00:43.800961 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.800966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.801213 (+ 247us) server_negotiation.cc:407] Connection header received
0620 15:00:43.801366 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.801369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.801411 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.801516 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:43.801784 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.802302 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.803405 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.803859 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.803951 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.803991 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.804263 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.804341 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:00:43.804493 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.800767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:50305 (local address 127.7.46.131:42993)
0620 15:00:43.800899 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.800904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.801152 (+ 248us) server_negotiation.cc:407] Connection header received
0620 15:00:43.801330 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.801333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.801388 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.801453 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:43.801789 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.802307 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.803463 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.803955 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.804077 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.804135 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.804321 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.804396 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:00:43.804492 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.800723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:36425 (local address 127.7.46.130:40737)
0620 15:00:43.801142 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.801153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.801259 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.801542 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.801548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.801567 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:43.801678 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.801682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.802415 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.802421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.803292 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.803301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.803969 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.803974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.804124 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.804149 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.804221 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.804302 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":24,"thread_start_us":71,"threads_started":1}
I20260620 15:00:43.804498 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.800695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:42993 (local address 127.7.46.130:50305)
0620 15:00:43.801043 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.801054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.801189 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.801484 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.801489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.801509 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:43.801673 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.801679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.802434 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.802437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.803316 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.803324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.804075 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.804078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.804180 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.804196 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.804290 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.804350 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":62,"thread_start_us":103,"threads_started":1}
I20260620 15:00:43.805225 22910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23feaacc9fb44fe8ae95521145494d5b" candidate_uuid: "f3d61096013345c38e38c07bbeac36bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378e968466b446ff8221671810afd5eb" is_pre_election: true
I20260620 15:00:43.805267 23178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23feaacc9fb44fe8ae95521145494d5b" candidate_uuid: "f3d61096013345c38e38c07bbeac36bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9186149806774e538dfd30894999724f" is_pre_election: true
I20260620 15:00:43.805531 23178 raft_consensus.cc:2454] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d61096013345c38e38c07bbeac36bc in term 0.
I20260620 15:00:43.805532 22910 raft_consensus.cc:2454] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d61096013345c38e38c07bbeac36bc in term 0.
I20260620 15:00:43.806025 22978 leader_election.cc:302] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9186149806774e538dfd30894999724f, f3d61096013345c38e38c07bbeac36bc; no voters:
I20260620 15:00:43.806298 23247 raft_consensus.cc:2781] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:43.806391 23247 raft_consensus.cc:482] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:43.806444 23247 raft_consensus.cc:3037] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:43.807657 23247 raft_consensus.cc:504] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.808032 23247 leader_election.cc:288] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [CANDIDATE]: Term 1 election: Requested vote from peers 9186149806774e538dfd30894999724f (127.7.46.131:42993), 378e968466b446ff8221671810afd5eb (127.7.46.129:36425)
I20260620 15:00:43.808389 22910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23feaacc9fb44fe8ae95521145494d5b" candidate_uuid: "f3d61096013345c38e38c07bbeac36bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378e968466b446ff8221671810afd5eb"
I20260620 15:00:43.808434 23178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23feaacc9fb44fe8ae95521145494d5b" candidate_uuid: "f3d61096013345c38e38c07bbeac36bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9186149806774e538dfd30894999724f"
I20260620 15:00:43.808543 22910 raft_consensus.cc:3037] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:43.808547 23178 raft_consensus.cc:3037] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:43.809552 23178 raft_consensus.cc:2454] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d61096013345c38e38c07bbeac36bc in term 1.
I20260620 15:00:43.809646 22910 raft_consensus.cc:2454] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d61096013345c38e38c07bbeac36bc in term 1.
I20260620 15:00:43.809867 22978 leader_election.cc:302] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9186149806774e538dfd30894999724f, f3d61096013345c38e38c07bbeac36bc; no voters:
I20260620 15:00:43.810021 23247 raft_consensus.cc:2781] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:43.810299 23247 raft_consensus.cc:686] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 1 LEADER]: Becoming Leader. State: Replica: f3d61096013345c38e38c07bbeac36bc, State: Running, Role: LEADER
I20260620 15:00:43.810626 23247 consensus_queue.cc:237] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:00:43.813845 22764 catalog_manager.cc:5495] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f3d61096013345c38e38c07bbeac36bc (127.7.46.130). New cstate: current_term: 1 leader_uuid: "f3d61096013345c38e38c07bbeac36bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:43.903546 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.899878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.7.46.130:34213)
0620 15:00:43.900019 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.900024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.900071 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:00:43.900271 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.900274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.900331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.900444 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:43.900713 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.901217 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.902092 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.902340 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.902454 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.902757 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:43.903170 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:43.903315 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.903348 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.903403 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:00:43.909242 22910 raft_consensus.cc:1261] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Refusing update from remote peer f3d61096013345c38e38c07bbeac36bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:43.909242 23178 raft_consensus.cc:1261] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 1 FOLLOWER]: Refusing update from remote peer f3d61096013345c38e38c07bbeac36bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:43.909948 23252 consensus_queue.cc:1035] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:43.910130 23247 consensus_queue.cc:1035] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:43.916800 23255 mvcc.cc:205] Tried to move back new op lower bound from 7298939469445328896 to 7298939469051109376. Current Snapshot: MvccSnapshot[applied={T|T < 7298939469445328896}]
I20260620 15:00:43.917908 23257 mvcc.cc:205] Tried to move back new op lower bound from 7298939469445328896 to 7298939469051109376. Current Snapshot: MvccSnapshot[applied={T|T < 7298939469445328896}]
I20260620 15:00:43.919576 23256 mvcc.cc:205] Tried to move back new op lower bound from 7298939469445328896 to 7298939469051109376. Current Snapshot: MvccSnapshot[applied={T|T < 7298939469445328896 or (T in {7298939469445328896})}]
I20260620 15:00:43.924686 22763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:43.924851 22763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:43.927104 22763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Sending DeleteTablet for 3 replicas of tablet 23feaacc9fb44fe8ae95521145494d5b
I20260620 15:00:43.927881 23158 tablet_service.cc:1514] Processing DeleteTablet for tablet 23feaacc9fb44fe8ae95521145494d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:43 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:55590
I20260620 15:00:43.927918 22890 tablet_service.cc:1514] Processing DeleteTablet for tablet 23feaacc9fb44fe8ae95521145494d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:43 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:45150
I20260620 15:00:43.928007 23024 tablet_service.cc:1514] Processing DeleteTablet for tablet 23feaacc9fb44fe8ae95521145494d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:43 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:44512
I20260620 15:00:43.928331 23265 tablet_replica.cc:330] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: stopping tablet replica
I20260620 15:00:43.928478 23266 tablet_replica.cc:330] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: stopping tablet replica
I20260620 15:00:43.928586 23265 raft_consensus.cc:2229] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:43.928731 23266 raft_consensus.cc:2229] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:43.928791 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 22827
I20260620 15:00:43.928849 23265 raft_consensus.cc:2258] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:43.931128 23265 ts_tablet_manager.cc:1905] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:43.931960 23267 tablet_replica.cc:330] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: stopping tablet replica
I20260620 15:00:43.932220 23267 raft_consensus.cc:2229] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:43.932551 23267 raft_consensus.cc:2258] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:43.933820 23267 ts_tablet_manager.cc:1905] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:43.934536 23265 ts_tablet_manager.cc:1918] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:43.934629 23265 log.cc:1196] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/23feaacc9fb44fe8ae95521145494d5b
I20260620 15:00:43.934958 23265 ts_tablet_manager.cc:1939] T 23feaacc9fb44fe8ae95521145494d5b P 9186149806774e538dfd30894999724f: Deleting consensus metadata
I20260620 15:00:43.936141 22750 catalog_manager.cc:4841] TS 9186149806774e538dfd30894999724f (127.7.46.131:42993): tablet 23feaacc9fb44fe8ae95521145494d5b (table test-table [id=257f2994fc4c4351b658dd3655cdee37]) successfully deleted
I20260620 15:00:43.936290 23267 ts_tablet_manager.cc:1918] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:43.936380 23267 log.cc:1196] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/23feaacc9fb44fe8ae95521145494d5b
I20260620 15:00:43.936668 23267 ts_tablet_manager.cc:1939] T 23feaacc9fb44fe8ae95521145494d5b P f3d61096013345c38e38c07bbeac36bc: Deleting consensus metadata
I20260620 15:00:43.937618 22750 catalog_manager.cc:4841] TS f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213): tablet 23feaacc9fb44fe8ae95521145494d5b (table test-table [id=257f2994fc4c4351b658dd3655cdee37]) successfully deleted
W20260620 15:00:43.937623 22751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:36425 (error 108)
I20260620 15:00:43.937888 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 22961
I20260620 15:00:43.939273 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.938914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:36425 (local address 127.0.0.1:45162)
0620 15:00:43.939091 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.939164 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:36425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 15:00:43.939422 22751 catalog_manager.cc:4584] TS 378e968466b446ff8221671810afd5eb (127.7.46.129:36425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23feaacc9fb44fe8ae95521145494d5b: Network error: Client connection negotiation failed: client connection to 127.7.46.129:36425: connect: Connection refused (error 111)
I20260620 15:00:43.943650 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 23095
I20260620 15:00:43.949064 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 22733
I20260620 15:01:03.955603 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43303
--webserver_interface=127.7.46.190
--webserver_port=45585
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:04.067032 23269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:04.067286 23269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:04.067389 23269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:04.071408 23269 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:01:04.071455 23269 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:01:04.071471 23269 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:01:04.071485 23269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:04.071504 23269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:04.071522 23269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:04.071537 23269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:04.071552 23269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:04.076660 23269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=45585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:04.077845 23269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:04.079093 23269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:04.092056 23278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:04.092095 23275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:04.092067 23276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:04.092396 23269 server_base.cc:1034] running on GCE node
I20260620 15:01:04.093104 23269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:04.094275 23269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:04.095471 23269 hybrid_clock.cc:648] HybridClock initialized: now 1781967664095443 us; error 51 us; skew 500 ppm
Jun 20 15:01:04 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:04.104952 23269 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:01:04.105854 23269 webserver.cc:460] Webserver started at http://127.7.46.190:45585/ using document root <none> and password file <none>
I20260620 15:01:04.106433 23269 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:04.106518 23269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:04.110522 23269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:01:04.112800 23285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:04.114112 23269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:01:04.114256 23269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "500bb606d97d499a8d5b1a233de20350"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "ca4a3ae3f3c87ec6fe8dc06c4f5dee49"
server_key_iv: "5fdbc11161247bcc92cfaded7c7507ce"
server_key_version: "encryptionkey@0"
I20260620 15:01:04.114885 23269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:04.131599 23269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:04.134691 23269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:04.134882 23269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:04.142338 23269 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:43303
I20260620 15:01:04.142683 23337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:43303 every 8 connection(s)
I20260620 15:01:04.143549 23269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:04.147939 23338 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:01:04.149464 23338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Bootstrap starting.
I20260620 15:01:04.152354 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 23269
I20260620 15:01:04.152766 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:36425
--local_ip_for_outbound_sockets=127.7.46.129
--tserver_master_addrs=127.7.46.190:43303
--webserver_port=35003
--webserver_interface=127.7.46.129
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:01:04.166108 23338 log.cc:824] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:04.174672 23338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:01:04.175028 23338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Bootstrap complete.
I20260620 15:01:04.179134 23338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:01:04.179534 23338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500bb606d97d499a8d5b1a233de20350, State: Initialized, Role: FOLLOWER
I20260620 15:01:04.180096 23338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:01:04.180238 23338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:04.180332 23338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:04.180446 23338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:01:04.182240 23338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:01:04.182575 23338 leader_election.cc:302] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500bb606d97d499a8d5b1a233de20350; no voters:
I20260620 15:01:04.182915 23338 leader_election.cc:288] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:01:04.182982 23342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:01:04.183228 23342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 2 LEADER]: Becoming Leader. State: Replica: 500bb606d97d499a8d5b1a233de20350, State: Running, Role: LEADER
I20260620 15:01:04.183561 23342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:01:04.184103 23338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:04.184496 23344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500bb606d97d499a8d5b1a233de20350. Latest consensus state: current_term: 2 leader_uuid: "500bb606d97d499a8d5b1a233de20350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } } }
I20260620 15:01:04.184608 23344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:04.184837 23343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "500bb606d97d499a8d5b1a233de20350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } } }
I20260620 15:01:04.184944 23343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:04.185071 23346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:04.189697 23346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:04.190433 23346 catalog_manager.cc:1259] Loaded cluster ID: 5716e8328a2a4a318a4f8d5ad34bec6d
I20260620 15:01:04.190493 23346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:04.192910 23346 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:04.193624 23346 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Loaded TSK: 0
I20260620 15:01:04.205175 23346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Generated new TSK 1
I20260620 15:01:04.205413 23346 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:01:04.263799 23340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:04.264081 23340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:04.264158 23340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:04.268143 23340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:04.268217 23340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:04.268303 23340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:01:04.273324 23340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:36425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=35003
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:04.274267 23340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:04.275218 23340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:04.285526 23365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:04.285521 23368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:04.285521 23366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:04.285722 23340 server_base.cc:1034] running on GCE node
I20260620 15:01:04.286224 23340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:04.286959 23340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:04.288151 23340 hybrid_clock.cc:648] HybridClock initialized: now 1781967664288139 us; error 30 us; skew 500 ppm
Jun 20 15:01:04 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:04.291254 23340 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:01:04.292158 23340 webserver.cc:460] Webserver started at http://127.7.46.129:35003/ using document root <none> and password file <none>
I20260620 15:01:04.292680 23340 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:04.292778 23340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:04.296454 23340 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:04.298640 23375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:04.299921 23340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:04.300050 23340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "378e968466b446ff8221671810afd5eb"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "a321b9c445d26902f2fe37c382a85d71"
server_key_iv: "8f67c9e15c53ca1e31cfdb7a2fdaa74b"
server_key_version: "encryptionkey@0"
I20260620 15:01:04.300552 23340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:04.313869 23340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:04.317014 23340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:04.317214 23340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:04.317768 23340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:04.318913 23382 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 15:01:04.322852 23340 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 15:01:04.322911 23340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 15:01:04.322962 23340 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 15:01:04.325047 23340 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 15:01:04.325102 23340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 15:01:04.325333 23382 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Bootstrap starting.
I20260620 15:01:04.335808 23340 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:36425
I20260620 15:01:04.335863 23489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:36425 every 8 connection(s)
I20260620 15:01:04.336943 23340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:04.338992 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 23340
I20260620 15:01:04.339490 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:34213
--local_ip_for_outbound_sockets=127.7.46.130
--tserver_master_addrs=127.7.46.190:43303
--webserver_port=40707
--webserver_interface=127.7.46.130
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:01:04.346872 23382 log.cc:824] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:01:04 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:04.355527 23382 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:01:04.355962 23382 tablet_bootstrap.cc:492] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Bootstrap complete.
I20260620 15:01:04.357334 23382 ts_tablet_manager.cc:1397] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.008s
I20260620 15:01:04.357777 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.339061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:56023 (local address 127.7.46.190:43303)
0620 15:01:04.339502 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.339524 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.344030 (+ 4506us) server_negotiation.cc:407] Connection header received
0620 15:01:04.345592 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.345609 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.346064 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.346462 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.347742 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.348747 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.349876 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.350329 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.350511 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.352841 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:04.352847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:04.352858 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:04.352885 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:04.354591 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.355110 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.355115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.355122 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.355234 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.355631 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.355635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.355636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.356081 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:04.356229 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.356731 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.356988 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":99,"threads_started":1}
I20260620 15:01:04.357791 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.339030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:56023)
0620 15:01:04.343808 (+ 4778us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.343868 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.345082 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.346545 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.346553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.347021 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:04.347555 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.347569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.348910 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.348913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.349751 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.349760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.350471 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.350475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.350653 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.351246 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:04.351262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:04.352653 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:04.354727 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:04.354732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:04.354744 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:04.354995 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:04.355337 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:04.355340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:04.355342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:04.355480 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:04.356204 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:04.356209 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:04.356435 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 15:01:04.356613 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 15:01:04.356918 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4471,"thread_start_us":128,"threads_started":1}
I20260620 15:01:04.360464 23490 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43303
I20260620 15:01:04.360270 23382 raft_consensus.cc:348] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.360700 23490 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:04.360769 23382 raft_consensus.cc:729] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378e968466b446ff8221671810afd5eb, State: Initialized, Role: FOLLOWER
I20260620 15:01:04.361223 23490 heartbeater.cc:511] Master 127.7.46.190:43303 requested a full tablet report, sending...
I20260620 15:01:04.361265 23382 consensus_queue.cc:260] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.362264 23382 ts_tablet_manager.cc:1428] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 15:01:04.363665 23302 ts_manager.cc:194] Registered new tserver with Master: 378e968466b446ff8221671810afd5eb (127.7.46.129:36425)
I20260620 15:01:04.366873 23302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:56023
I20260620 15:01:04.368916 23490 heartbeater.cc:503] Master 127.7.46.190:43303 was elected leader, sending a full tablet report...
Jun 20 15:01:04 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.129@KRBTEST.COM
I20260620 15:01:04.378165 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.366188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:36425 (local address 127.0.0.1:42320)
0620 15:01:04.366884 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.366925 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.367045 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.367626 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.367631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.369541 (+ 1910us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:04.369661 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.369669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.371008 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.371011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.371704 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.371714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.372527 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.372529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.372609 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.373168 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:04.373179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:04.374553 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:04.376632 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:04.376640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:04.376643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:04.376853 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:04.377158 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:04.377160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:04.377162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:04.377232 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:04.377699 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:04.377704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:04.377796 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 15:01:04.377905 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:01:04.378036 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":84,"threads_started":1}
I20260620 15:01:04.378326 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.366306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.7.46.129:36425)
0620 15:01:04.366670 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.366681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.367065 (+ 384us) server_negotiation.cc:407] Connection header received
0620 15:01:04.367187 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.367192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.367373 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.367502 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.369789 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.370846 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.371938 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.372405 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.372514 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.374684 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:04.374693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:04.374704 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:04.374729 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:04.376525 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.376968 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.376974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.376975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.377066 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.377314 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.377318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.377319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.377593 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:04.377719 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.378014 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.378169 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":108,"threads_started":1}
I20260620 15:01:04.379169 23406 tablet_service.cc:1514] Processing DeleteTablet for tablet 23feaacc9fb44fe8ae95521145494d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:43 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:42320
I20260620 15:01:04.379976 23499 tablet_replica.cc:330] stopping tablet replica
I20260620 15:01:04.380247 23499 raft_consensus.cc:2229] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:04.380486 23499 raft_consensus.cc:2258] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:04.381366 23499 ts_tablet_manager.cc:1905] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:04.385725 23499 ts_tablet_manager.cc:1918] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:04.385802 23499 log.cc:1196] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal/wals/23feaacc9fb44fe8ae95521145494d5b
I20260620 15:01:04.386080 23499 ts_tablet_manager.cc:1939] T 23feaacc9fb44fe8ae95521145494d5b P 378e968466b446ff8221671810afd5eb: Deleting consensus metadata
I20260620 15:01:04.387681 23288 catalog_manager.cc:4841] TS 378e968466b446ff8221671810afd5eb (127.7.46.129:36425): tablet 23feaacc9fb44fe8ae95521145494d5b (table test-table [id=257f2994fc4c4351b658dd3655cdee37]) successfully deleted
W20260620 15:01:04.456094 23494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:04.456368 23494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:04.456451 23494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:04.460395 23494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:04.460480 23494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:04.460566 23494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:01:04.465695 23494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:34213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=40707
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:04.466717 23494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:04.467787 23494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:04.477249 23507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:04.477276 23505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:04.477464 23494 server_base.cc:1034] running on GCE node
W20260620 15:01:04.477413 23504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:04.478070 23494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:04.478628 23494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:04.479872 23494 hybrid_clock.cc:648] HybridClock initialized: now 1781967664479849 us; error 26 us; skew 500 ppm
Jun 20 15:01:04 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:04.482625 23494 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:01:04.483520 23494 webserver.cc:460] Webserver started at http://127.7.46.130:40707/ using document root <none> and password file <none>
I20260620 15:01:04.484138 23494 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:04.484241 23494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:04.488137 23494 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:04.490118 23514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:04.491315 23494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:04.491453 23494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "f3d61096013345c38e38c07bbeac36bc"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "a8db5ab978612184d79659ef2e059627"
server_key_iv: "c7da6e27c2f57a01ed5f0e19abc27d61"
server_key_version: "encryptionkey@0"
I20260620 15:01:04.492055 23494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:04.501467 23494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:04.504644 23494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:04.504830 23494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:04.505431 23494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:04.506210 23494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:04.506279 23494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:04.506348 23494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:04.506395 23494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:04.515046 23494 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:34213
I20260620 15:01:04.515069 23627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:34213 every 8 connection(s)
I20260620 15:01:04.516007 23494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
Jun 20 15:01:04 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:04.525722 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 23494
I20260620 15:01:04.526126 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:42993
--local_ip_for_outbound_sockets=127.7.46.131
--tserver_master_addrs=127.7.46.190:43303
--webserver_port=42929
--webserver_interface=127.7.46.131
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:01:04.529510 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.517679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:39827 (local address 127.7.46.190:43303)
0620 15:01:04.517814 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.517820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.518609 (+ 789us) server_negotiation.cc:407] Connection header received
0620 15:01:04.520033 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.520037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.520101 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.520197 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.521308 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.521875 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.523005 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.523253 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.523366 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.525801 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:04.525804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:04.525806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:04.525854 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:04.527441 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.528011 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.528015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.528017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.528103 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.528439 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.528442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.528443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.528624 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:04.528728 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.529195 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.529325 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:01:04.530274 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.517967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:39827)
0620 15:01:04.518471 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.518511 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.519815 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.520365 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.520373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.520700 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:04.521161 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.521174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.521999 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.522003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.522879 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.522888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.523399 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.523404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.523574 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.524165 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:04.524180 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:04.525631 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:04.527594 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:04.527599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:04.527609 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:04.527909 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:04.528197 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:04.528201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:04.528203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:04.528332 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:04.528727 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:04.528733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:04.528965 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 15:01:04.529138 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 15:01:04.529437 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
I20260620 15:01:04.531422 23628 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43303
I20260620 15:01:04.531683 23628 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:04.532323 23628 heartbeater.cc:511] Master 127.7.46.190:43303 requested a full tablet report, sending...
I20260620 15:01:04.533491 23302 ts_manager.cc:194] Registered new tserver with Master: f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213)
I20260620 15:01:04.534358 23302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:39827
W20260620 15:01:04.635380 23631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:04.635624 23631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:04.635666 23631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:04.639647 23631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:04.639698 23631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:04.639751 23631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:01:04.644735 23631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:42993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=42929
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:04.645641 23631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:04.646579 23631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:04.655849 23636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:04.655846 23637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:04.655846 23639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:04.656031 23631 server_base.cc:1034] running on GCE node
I20260620 15:01:04.656459 23631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:04.657058 23631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:04.658209 23631 hybrid_clock.cc:648] HybridClock initialized: now 1781967664658175 us; error 47 us; skew 500 ppm
Jun 20 15:01:04 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:04.660787 23631 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:01:04.661593 23631 webserver.cc:460] Webserver started at http://127.7.46.131:42929/ using document root <none> and password file <none>
I20260620 15:01:04.662094 23631 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:04.662160 23631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:04.665707 23631 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:04.667580 23646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:04.668799 23631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:04.668911 23631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "9186149806774e538dfd30894999724f"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "c63efda3cc419b184a492bb5fb065fd6"
server_key_iv: "373369966de598371c39538de99e9809"
server_key_version: "encryptionkey@0"
I20260620 15:01:04.669399 23631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:04.700048 23631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:04.702728 23631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:04.702915 23631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:04.703539 23631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:04.704418 23631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:04.704514 23631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:04.704581 23631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:04.704617 23631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:04.713176 23631 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:42993
I20260620 15:01:04.713205 23759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:42993 every 8 connection(s)
I20260620 15:01:04.714078 23631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:04.722713 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 23631
Jun 20 15:01:04 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967664, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:04.727232 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.715752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:38167 (local address 127.7.46.190:43303)
0620 15:01:04.715898 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.715901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.717130 (+ 1229us) server_negotiation.cc:407] Connection header received
0620 15:01:04.718020 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.718023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.718067 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.718137 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.719352 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.719923 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.721000 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.721206 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.721279 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.723535 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:04.723539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:04.723542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:04.723568 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:04.725228 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.725680 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.725684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.725686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.725775 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.726126 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.726128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.726129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.726289 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:04.726405 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.726955 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.727063 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:01:04.728118 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.716148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:38167)
0620 15:01:04.716983 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.717022 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.717765 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.718347 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.718354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.718679 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:04.719193 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.719209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.720031 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.720034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.720901 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.720911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.721325 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.721328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.721454 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.721930 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:04.721945 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:04.723363 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:04.725334 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:04.725338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:04.725348 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:04.725591 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:04.725864 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:04.725867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:04.725868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:04.726032 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:04.726429 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:04.726435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:04.726711 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 15:01:04.726904 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 15:01:04.727131 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":513,"threads_started":1}
I20260620 15:01:04.729266 23760 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43303
I20260620 15:01:04.729530 23760 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:04.730093 23760 heartbeater.cc:511] Master 127.7.46.190:43303 requested a full tablet report, sending...
I20260620 15:01:04.731302 23302 ts_manager.cc:194] Registered new tserver with Master: 9186149806774e538dfd30894999724f (127.7.46.131:42993)
I20260620 15:01:04.732024 23302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:38167
I20260620 15:01:04.732551 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.723364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.7.46.190:43303)
0620 15:01:04.723616 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.723619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.723926 (+ 307us) server_negotiation.cc:407] Connection header received
0620 15:01:04.724096 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.724099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.724148 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.724247 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.725051 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.725737 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.726869 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.727195 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.727535 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.728860 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:04.728864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:04.728866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:04.728895 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:04.730466 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.731026 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.731031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.731034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.731121 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.731602 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.731605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.731606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.731805 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:04.731966 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.732220 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.732351 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
I20260620 15:01:04.734336 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:01:04.737984 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.734982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.7.46.190:43303)
0620 15:01:04.735099 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.735103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.735215 (+ 112us) server_negotiation.cc:407] Connection header received
0620 15:01:04.735369 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.735371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.735452 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.735531 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:04.735692 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.736241 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.737028 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.737243 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.737360 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.737593 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:04.737704 (+ 111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 15:01:04.737900 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43852: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 15:01:04.738065 23764 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43852: token expired
W20260620 15:01:04.738368 23229 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.46.190:43303)
I20260620 15:01:04.744949 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.738851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.7.46.190:43303)
0620 15:01:04.738947 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.738951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.738962 (+ 11us) server_negotiation.cc:407] Connection header received
0620 15:01:04.738991 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.738993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.739025 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.739092 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.739606 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.740127 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.740865 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.741072 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.741154 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.742283 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:04.742286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:04.742288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:04.742310 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:04.743622 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.744030 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.744032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.744033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.744112 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:04.744320 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:04.744322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:04.744323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:04.744472 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:04.744570 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.744718 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.744845 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 15:01:04.747649 23302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:01:04.752719 23302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:01:04.765558 23406 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0376f19d8d4f59815abaa78c412c7a (DEFAULT_TABLE table=test-table [id=7522664bc26d4db6af45c139e7908c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:04.766160 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0376f19d8d4f59815abaa78c412c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:04.768877 23382 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Bootstrap starting.
I20260620 15:01:04.769635 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.764341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:34213 (local address 127.0.0.1:46578)
0620 15:01:04.764464 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.764481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.764567 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.765764 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.765767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.765784 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:04.765889 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.765899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.766961 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.766965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.768178 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.768188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.769208 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.769212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.769308 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.769352 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 15:01:04.769411 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 15:01:04.769508 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:01:04.769716 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.764556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.7.46.130:34213)
0620 15:01:04.765156 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.765171 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.765197 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:01:04.765285 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.765303 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.765530 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.765783 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:04.765976 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.766835 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.768333 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.769086 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.769214 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.769377 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.769456 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.769550 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":107,"threads_started":1}
I20260620 15:01:04.769829 23382 tablet_bootstrap.cc:654] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:04.771870 23382 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: No bootstrap required, opened a new log
I20260620 15:01:04.771967 23382 ts_tablet_manager.cc:1397] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 15:01:04.772356 23382 raft_consensus.cc:348] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.772496 23382 raft_consensus.cc:374] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:04.772529 23382 raft_consensus.cc:729] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378e968466b446ff8221671810afd5eb, State: Initialized, Role: FOLLOWER
I20260620 15:01:04.772691 23382 consensus_queue.cc:260] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.772992 23382 ts_tablet_manager.cc:1428] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 15:01:04.772850 23562 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0376f19d8d4f59815abaa78c412c7a (DEFAULT_TABLE table=test-table [id=7522664bc26d4db6af45c139e7908c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:04.773465 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.764569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:42993 (local address 127.0.0.1:50516)
0620 15:01:04.764790 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.764804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.764886 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.769519 (+ 4633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.769522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.769538 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:04.769614 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.769619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.770803 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.770808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.771932 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.771944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.773095 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.773099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.773197 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.773221 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:01:04.773268 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 15:01:04.773332 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260620 15:01:04.773972 23562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0376f19d8d4f59815abaa78c412c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:04.774137 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.764996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.7.46.131:42993)
0620 15:01:04.769056 (+ 4060us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.769068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.769088 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:01:04.769162 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.769169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.769360 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.769764 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:04.769931 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.770673 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.772085 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.772944 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.773726 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.773867 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.773915 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.774001 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3948,"thread_start_us":166,"threads_started":1}
I20260620 15:01:04.776821 23694 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0376f19d8d4f59815abaa78c412c7a (DEFAULT_TABLE table=test-table [id=7522664bc26d4db6af45c139e7908c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:04.777817 23694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0376f19d8d4f59815abaa78c412c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:04.781800 23773 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: Bootstrap starting.
I20260620 15:01:04.784363 23773 tablet_bootstrap.cc:654] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:04.785027 23774 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Bootstrap starting.
I20260620 15:01:04.785223 23773 log.cc:824] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:04.787266 23773 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: No bootstrap required, opened a new log
I20260620 15:01:04.787494 23773 ts_tablet_manager.cc:1397] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 15:01:04.787592 23774 tablet_bootstrap.cc:654] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:04.788388 23774 log.cc:824] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:04.790457 23774 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: No bootstrap required, opened a new log
I20260620 15:01:04.790637 23774 ts_tablet_manager.cc:1397] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 15:01:04.791339 23773 raft_consensus.cc:348] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.791611 23773 raft_consensus.cc:374] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:04.791668 23773 raft_consensus.cc:729] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d61096013345c38e38c07bbeac36bc, State: Initialized, Role: FOLLOWER
I20260620 15:01:04.792135 23773 consensus_queue.cc:260] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.792737 23628 heartbeater.cc:503] Master 127.7.46.190:43303 was elected leader, sending a full tablet report...
I20260620 15:01:04.793139 23773 ts_tablet_manager.cc:1428] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 15:01:04.794162 23774 raft_consensus.cc:348] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.794353 23774 raft_consensus.cc:374] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:04.794441 23774 raft_consensus.cc:729] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9186149806774e538dfd30894999724f, State: Initialized, Role: FOLLOWER
I20260620 15:01:04.794992 23774 consensus_queue.cc:260] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.795755 23760 heartbeater.cc:503] Master 127.7.46.190:43303 was elected leader, sending a full tablet report...
I20260620 15:01:04.796105 23774 ts_tablet_manager.cc:1428] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
W20260620 15:01:04.838227 23491 tablet.cc:2369] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:04.912209 23777 raft_consensus.cc:482] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:04.912438 23777 raft_consensus.cc:504] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.913669 23777 leader_election.cc:288] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9186149806774e538dfd30894999724f (127.7.46.131:42993), 378e968466b446ff8221671810afd5eb (127.7.46.129:36425)
I20260620 15:01:04.918500 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.913975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:46531 (local address 127.7.46.131:42993)
0620 15:01:04.914113 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.914118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.914134 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:01:04.914193 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.914197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.914282 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.914375 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:04.914703 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.915496 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.917183 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.917799 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.917936 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.918010 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.918284 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.918370 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:01:04.918504 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.914130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:40101 (local address 127.7.46.129:36425)
0620 15:01:04.914403 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.914407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.914443 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:01:04.914572 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.914574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.914625 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.914757 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:04.915010 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.915699 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.917183 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.917847 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.917951 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.918129 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.918283 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.918360 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260620 15:01:04.918529 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.913862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:42993 (local address 127.7.46.130:46531)
0620 15:01:04.913977 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.913994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.914089 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.914414 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.914420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.914442 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:04.914595 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.914601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.915622 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.915625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.917050 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.917061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.917970 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.917974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.918116 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.918177 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 15:01:04.918257 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:01:04.918379 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":115,"spinlock_wait_cycles":5120}
I20260620 15:01:04.918969 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.913932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:36425 (local address 127.7.46.130:40101)
0620 15:01:04.914371 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.914387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.914473 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.914775 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.914777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.914793 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:04.914891 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.914895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.915806 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.915810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.917050 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.917061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.917970 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.917981 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.918112 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.918177 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 15:01:04.918257 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:01:04.918394 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":129,"spinlock_wait_cycles":384,"thread_start_us":118,"threads_started":1}
I20260620 15:01:04.919305 23714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0376f19d8d4f59815abaa78c412c7a" candidate_uuid: "f3d61096013345c38e38c07bbeac36bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9186149806774e538dfd30894999724f" is_pre_election: true
I20260620 15:01:04.919579 23444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0376f19d8d4f59815abaa78c412c7a" candidate_uuid: "f3d61096013345c38e38c07bbeac36bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378e968466b446ff8221671810afd5eb" is_pre_election: true
I20260620 15:01:04.919648 23714 raft_consensus.cc:2454] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d61096013345c38e38c07bbeac36bc in term 0.
I20260620 15:01:04.919872 23444 raft_consensus.cc:2454] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d61096013345c38e38c07bbeac36bc in term 0.
I20260620 15:01:04.920286 23516 leader_election.cc:302] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 378e968466b446ff8221671810afd5eb, 9186149806774e538dfd30894999724f, f3d61096013345c38e38c07bbeac36bc; no voters:
I20260620 15:01:04.920574 23777 raft_consensus.cc:2781] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:01:04.920650 23777 raft_consensus.cc:482] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:01:04.920679 23777 raft_consensus.cc:3037] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:04.921695 23777 raft_consensus.cc:504] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.922070 23777 leader_election.cc:288] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [CANDIDATE]: Term 1 election: Requested vote from peers 9186149806774e538dfd30894999724f (127.7.46.131:42993), 378e968466b446ff8221671810afd5eb (127.7.46.129:36425)
I20260620 15:01:04.922400 23444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0376f19d8d4f59815abaa78c412c7a" candidate_uuid: "f3d61096013345c38e38c07bbeac36bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378e968466b446ff8221671810afd5eb"
I20260620 15:01:04.922439 23714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0376f19d8d4f59815abaa78c412c7a" candidate_uuid: "f3d61096013345c38e38c07bbeac36bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9186149806774e538dfd30894999724f"
I20260620 15:01:04.922544 23444 raft_consensus.cc:3037] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:04.922567 23714 raft_consensus.cc:3037] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:04.923487 23444 raft_consensus.cc:2454] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d61096013345c38e38c07bbeac36bc in term 1.
I20260620 15:01:04.923630 23714 raft_consensus.cc:2454] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d61096013345c38e38c07bbeac36bc in term 1.
I20260620 15:01:04.923795 23517 leader_election.cc:302] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378e968466b446ff8221671810afd5eb, f3d61096013345c38e38c07bbeac36bc; no voters:
I20260620 15:01:04.923966 23777 raft_consensus.cc:2781] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:04.924196 23777 raft_consensus.cc:686] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 1 LEADER]: Becoming Leader. State: Replica: f3d61096013345c38e38c07bbeac36bc, State: Running, Role: LEADER
I20260620 15:01:04.924530 23777 consensus_queue.cc:237] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:04.927621 23301 catalog_manager.cc:5495] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f3d61096013345c38e38c07bbeac36bc (127.7.46.130). New cstate: current_term: 1 leader_uuid: "f3d61096013345c38e38c07bbeac36bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } health_report { overall_health: UNKNOWN } } }
I20260620 15:01:04.961825 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.958291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.7.46.130:34213)
0620 15:01:04.958449 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.958453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.958468 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:01:04.958648 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.958650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.958706 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.958820 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:04.959054 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.959557 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.960401 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.960628 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.960739 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.961048 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:04.961432 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:01:04.961565 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:04.961628 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 15:01:04.961682 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 15:01:04.965000 23761 tablet.cc:2369] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:04.967549 23714 raft_consensus.cc:1261] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 1 FOLLOWER]: Refusing update from remote peer f3d61096013345c38e38c07bbeac36bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:04.967547 23444 raft_consensus.cc:1261] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Refusing update from remote peer f3d61096013345c38e38c07bbeac36bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:04.968273 23781 consensus_queue.cc:1035] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:04.968438 23777 consensus_queue.cc:1035] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:04.974475 23783 mvcc.cc:205] Tried to move back new op lower bound from 7298939555700482048 to 7298939555533303808. Current Snapshot: MvccSnapshot[applied={T|T < 7298939555700482048}]
I20260620 15:01:04.974522 23784 mvcc.cc:205] Tried to move back new op lower bound from 7298939555700482048 to 7298939555533303808. Current Snapshot: MvccSnapshot[applied={T|T < 7298939555700482048}]
I20260620 15:01:04.982722 23302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:04.982885 23302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:04.985224 23302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Sending DeleteTablet for 3 replicas of tablet 9d0376f19d8d4f59815abaa78c412c7a
I20260620 15:01:04.985992 23406 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d0376f19d8d4f59815abaa78c412c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:04 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:42320
I20260620 15:01:04.985996 23562 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d0376f19d8d4f59815abaa78c412c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:04 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:46578
I20260620 15:01:04.986393 23793 tablet_replica.cc:330] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: stopping tablet replica
I20260620 15:01:04.986429 23694 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d0376f19d8d4f59815abaa78c412c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:04 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:50516
I20260620 15:01:04.986716 23793 raft_consensus.cc:2229] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:01:04.986845 23794 tablet_replica.cc:330] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: stopping tablet replica
I20260620 15:01:04.987066 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 23340
I20260620 15:01:04.987087 23793 raft_consensus.cc:2258] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:04.987787 23794 raft_consensus.cc:2229] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:04.988444 23794 raft_consensus.cc:2258] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:04.989017 23793 ts_tablet_manager.cc:1905] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:04.989675 23794 ts_tablet_manager.cc:1905] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:04.991261 23793 ts_tablet_manager.cc:1918] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:04.991369 23793 log.cc:1196] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/9d0376f19d8d4f59815abaa78c412c7a
I20260620 15:01:04.991698 23793 ts_tablet_manager.cc:1939] T 9d0376f19d8d4f59815abaa78c412c7a P f3d61096013345c38e38c07bbeac36bc: Deleting consensus metadata
I20260620 15:01:04.992117 23794 ts_tablet_manager.cc:1918] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:04.992205 23794 log.cc:1196] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/9d0376f19d8d4f59815abaa78c412c7a
I20260620 15:01:04.992547 23794 ts_tablet_manager.cc:1939] T 9d0376f19d8d4f59815abaa78c412c7a P 9186149806774e538dfd30894999724f: Deleting consensus metadata
I20260620 15:01:04.992766 23287 catalog_manager.cc:4841] TS f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213): tablet 9d0376f19d8d4f59815abaa78c412c7a (table test-table [id=7522664bc26d4db6af45c139e7908c34]) successfully deleted
I20260620 15:01:04.993714 23287 catalog_manager.cc:4841] TS 9186149806774e538dfd30894999724f (127.7.46.131:42993): tablet 9d0376f19d8d4f59815abaa78c412c7a (table test-table [id=7522664bc26d4db6af45c139e7908c34]) successfully deleted
W20260620 15:01:04.994805 23288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:36425 (error 108)
I20260620 15:01:04.995083 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 23494
I20260620 15:01:04.996114 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.995759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:36425 (local address 127.0.0.1:42324)
0620 15:01:04.995919 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.996012 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:36425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 15:01:04.996268 23288 catalog_manager.cc:4584] TS 378e968466b446ff8221671810afd5eb (127.7.46.129:36425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d0376f19d8d4f59815abaa78c412c7a: Network error: Client connection negotiation failed: client connection to 127.7.46.129:36425: connect: Connection refused (error 111)
I20260620 15:01:05.001417 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 23631
I20260620 15:01:05.006786 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 23269
I20260620 15:01:25.014384 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43303
--webserver_interface=127.7.46.190
--webserver_port=45585
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:25.121417 23796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:25.121656 23796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:25.121694 23796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:25.125756 23796 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:01:25.125800 23796 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:01:25.125816 23796 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:01:25.125830 23796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:25.125890 23796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:25.125926 23796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:25.125960 23796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:25.125998 23796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:25.131500 23796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=45585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:25.132499 23796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:25.133495 23796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:01:25.142071 23796 server_base.cc:1034] running on GCE node
W20260620 15:01:25.142148 23804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.142132 23801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.142279 23802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:25.142843 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:25.143898 23796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:25.145053 23796 hybrid_clock.cc:648] HybridClock initialized: now 1781967685145040 us; error 25 us; skew 500 ppm
Jun 20 15:01:25 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:25.150905 23796 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:01:25.151927 23796 webserver.cc:460] Webserver started at http://127.7.46.190:45585/ using document root <none> and password file <none>
I20260620 15:01:25.152496 23796 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:25.152586 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:25.156472 23796 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:01:25.158475 23811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.159655 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:25.159819 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "500bb606d97d499a8d5b1a233de20350"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "ca4a3ae3f3c87ec6fe8dc06c4f5dee49"
server_key_iv: "5fdbc11161247bcc92cfaded7c7507ce"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.160359 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:25.176034 23796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:25.178665 23796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:25.178865 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:25.185948 23796 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:43303
I20260620 15:01:25.186223 23863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:43303 every 8 connection(s)
I20260620 15:01:25.186978 23796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:25.190333 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 23796
I20260620 15:01:25.190837 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:36425
--local_ip_for_outbound_sockets=127.7.46.129
--tserver_master_addrs=127.7.46.190:43303
--webserver_port=35003
--webserver_interface=127.7.46.129
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:01:25.191368 23864 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:01:25.193480 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Bootstrap starting.
I20260620 15:01:25.210583 23864 log.cc:824] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:25.221529 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:01:25.221903 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Bootstrap complete.
I20260620 15:01:25.225983 23864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:01:25.226344 23864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500bb606d97d499a8d5b1a233de20350, State: Initialized, Role: FOLLOWER
I20260620 15:01:25.226855 23864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:01:25.226971 23864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:25.227057 23864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:25.227170 23864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:01:25.229101 23864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:01:25.229435 23864 leader_election.cc:302] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500bb606d97d499a8d5b1a233de20350; no voters:
I20260620 15:01:25.229718 23864 leader_election.cc:288] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:01:25.229851 23868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:01:25.230093 23868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [term 3 LEADER]: Becoming Leader. State: Replica: 500bb606d97d499a8d5b1a233de20350, State: Running, Role: LEADER
I20260620 15:01:25.230471 23868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } }
I20260620 15:01:25.230912 23864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:25.231839 23869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "500bb606d97d499a8d5b1a233de20350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } } }
I20260620 15:01:25.231973 23869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:25.232333 23870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500bb606d97d499a8d5b1a233de20350. Latest consensus state: current_term: 3 leader_uuid: "500bb606d97d499a8d5b1a233de20350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bb606d97d499a8d5b1a233de20350" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43303 } } }
I20260620 15:01:25.232477 23870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:25.232817 23877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:25.237614 23877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:25.238266 23877 catalog_manager.cc:1259] Loaded cluster ID: 5716e8328a2a4a318a4f8d5ad34bec6d
I20260620 15:01:25.238327 23877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:25.240845 23877 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:25.241552 23877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Loaded TSK: 0
I20260620 15:01:25.241632 23877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 500bb606d97d499a8d5b1a233de20350: Loaded TSK: 1
F20260620 15:01:25.242363 23877 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f316184e9ad google::LogMessage::Fail() at ??:0
@ 0x7f316185188a google::LogMessage::SendToLog() at ??:0
@ 0x7f316184e4de google::LogMessage::Flush() at ??:0
@ 0x7f316184fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f316451218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f316450af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f316c1b37bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f316c18d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f316c18e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f316c1cfdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f316c1ee82b std::function<>::operator()() at ??:0
@ 0x7f316c18db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f316c18ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f316c18b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f316c1cf640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3163aa03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3163aa0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3163aa23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3163a91512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f31650d56db start_thread at ??:0
@ 0x7f316034c61f clone at ??:0
W20260620 15:01:25.333979 23866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:25.334198 23866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:25.334240 23866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:25.338207 23866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:25.338254 23866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:25.338340 23866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:01:25.343451 23866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:36425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=35003
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:25.344455 23866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:25.345438 23866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:25.354745 23912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.354713 23911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.355080 23914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:25.355157 23866 server_base.cc:1034] running on GCE node
I20260620 15:01:25.355615 23866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:25.356285 23866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:25.357487 23866 hybrid_clock.cc:648] HybridClock initialized: now 1781967685357469 us; error 39 us; skew 500 ppm
Jun 20 15:01:25 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:25.360214 23866 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:01:25.361025 23866 webserver.cc:460] Webserver started at http://127.7.46.129:35003/ using document root <none> and password file <none>
I20260620 15:01:25.361558 23866 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:25.361647 23866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:25.365351 23866 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:25.367457 23921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.368772 23866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:25.368896 23866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "378e968466b446ff8221671810afd5eb"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-30t4"
server_key: "a321b9c445d26902f2fe37c382a85d71"
server_key_iv: "8f67c9e15c53ca1e31cfdb7a2fdaa74b"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.369400 23866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:25.393321 23866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:25.396452 23866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:25.396826 23866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:25.397478 23866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:25.398556 23928 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 15:01:25.403699 23866 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 15:01:25.403810 23866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 15:01:25.403905 23866 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 15:01:25.405511 23866 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 15:01:25.405601 23866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 15:01:25.405848 23928 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Bootstrap starting.
I20260620 15:01:25.416055 23866 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:36425
I20260620 15:01:25.416200 24035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:36425 every 8 connection(s)
I20260620 15:01:25.417014 23866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:25.417591 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 23866
I20260620 15:01:25.418004 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:34213
--local_ip_for_outbound_sockets=127.7.46.130
--tserver_master_addrs=127.7.46.190:43303
--webserver_port=40707
--webserver_interface=127.7.46.130
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:01:25.427647 23928 log.cc:824] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:25.436415 23928 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:01:25.436808 23928 tablet_bootstrap.cc:492] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Bootstrap complete.
I20260620 15:01:25.438182 23928 ts_tablet_manager.cc:1397] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.008s
I20260620 15:01:25.441406 23928 raft_consensus.cc:348] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:25.441929 23928 raft_consensus.cc:729] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378e968466b446ff8221671810afd5eb, State: Initialized, Role: FOLLOWER
I20260620 15:01:25.442463 23928 consensus_queue.cc:260] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:25.443455 23928 ts_tablet_manager.cc:1428] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 15:01:25.458495 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:25.420453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:51391)
0620 15:01:25.421268 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.421323 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:25.422332 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:25.457792 (+ 35460us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":140,"threads_started":1}
W20260620 15:01:25.458998 24036 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:43303 (0 consecutive failures): Network error: Failed to ping master at 127.7.46.190:43303: Client connection negotiation failed: client connection to 127.7.46.190:43303: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:01:25.459757 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.459505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:34749)
0620 15:01:25.459638 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.459697 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:01:25.460481 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.460279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:33445)
0620 15:01:25.460387 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.460437 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:01:25.460649 24036 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:01:25.535480 24038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:25.535739 24038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:25.535837 24038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:25.539712 24038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:25.539817 24038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:25.539913 24038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:01:25.545034 24038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:34213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=40707
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:25.545989 24038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:25.546947 24038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:25.556226 24046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.556494 24049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.556226 24047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:25.556890 24038 server_base.cc:1034] running on GCE node
I20260620 15:01:25.557277 24038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:25.557905 24038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:25.559108 24038 hybrid_clock.cc:648] HybridClock initialized: now 1781967685559090 us; error 28 us; skew 500 ppm
Jun 20 15:01:25 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:25.561815 24038 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:01:25.562641 24038 webserver.cc:460] Webserver started at http://127.7.46.130:40707/ using document root <none> and password file <none>
I20260620 15:01:25.563189 24038 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:25.563284 24038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:25.566846 24038 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:25.568858 24056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.570050 24038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:01:25.570210 24038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "f3d61096013345c38e38c07bbeac36bc"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "a8db5ab978612184d79659ef2e059627"
server_key_iv: "c7da6e27c2f57a01ed5f0e19abc27d61"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.570724 24038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:25.594619 24038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:25.597903 24038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:25.598112 24038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:25.598697 24038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:25.599516 24038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:25.599589 24038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.599656 24038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:25.599709 24038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.608415 24038 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:34213
I20260620 15:01:25.608454 24169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:34213 every 8 connection(s)
I20260620 15:01:25.609378 24038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:25.612998 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.611347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:40007)
0620 15:01:25.611811 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.612313 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
W20260620 15:01:25.613408 24170 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:43303 (0 consecutive failures): Network error: Failed to ping master at 127.7.46.190:43303: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
I20260620 15:01:25.613976 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.613762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:55517)
0620 15:01:25.613870 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.613923 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:01:25.614573 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.614370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:59273)
0620 15:01:25.614473 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.614517 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 15:01:25.614795 24170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:01:25.616173 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 24038
I20260620 15:01:25.616541 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:42993
--local_ip_for_outbound_sockets=127.7.46.131
--tserver_master_addrs=127.7.46.190:43303
--webserver_port=42929
--webserver_interface=127.7.46.131
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:25.723168 24173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:25.723383 24173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:25.723423 24173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:25.727453 24173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:25.727504 24173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:25.727602 24173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:01:25.732692 24173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:42993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=42929
--tserver_master_addrs=127.7.46.190:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:25.733598 24173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:25.734510 24173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:25.743319 24181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.743347 24179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.743355 24178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:25.743386 24173 server_base.cc:1034] running on GCE node
I20260620 15:01:25.744042 24173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:25.744596 24173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:25.745764 24173 hybrid_clock.cc:648] HybridClock initialized: now 1781967685745738 us; error 38 us; skew 500 ppm
Jun 20 15:01:25 dist-test-slave-30t4 krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:25.748431 24173 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:01:25.749195 24173 webserver.cc:460] Webserver started at http://127.7.46.131:42929/ using document root <none> and password file <none>
I20260620 15:01:25.749714 24173 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:25.749802 24173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:25.753545 24173 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:25.755478 24188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.756668 24173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:25.756805 24173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "9186149806774e538dfd30894999724f"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-30t4"
server_key: "c63efda3cc419b184a492bb5fb065fd6"
server_key_iv: "373369966de598371c39538de99e9809"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.757308 24173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:25.772146 24173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:25.774967 24173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:25.775177 24173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:25.775710 24173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:25.776513 24173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:25.776585 24173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.776654 24173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:25.776698 24173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.785344 24173 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:42993
I20260620 15:01:25.785364 24301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:42993 every 8 connection(s)
I20260620 15:01:25.786258 24173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:25.789731 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.788202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:36665)
0620 15:01:25.788615 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.789079 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
W20260620 15:01:25.790161 24302 heartbeater.cc:644] Failed to heartbeat to 127.7.46.190:43303 (0 consecutive failures): Network error: Failed to ping master at 127.7.46.190:43303: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
I20260620 15:01:25.790767 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.790556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:45985)
0620 15:01:25.790662 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.790711 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 15:01:25.791313 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.791109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:44289)
0620 15:01:25.791211 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.791253 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 15:01:25.791483 24302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:01:25.792157 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 24173
I20260620 15:01:26.461727 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.461205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:53557)
0620 15:01:26.461572 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.461625 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":88,"threads_started":1}
I20260620 15:01:26.615664 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.615311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:48493)
0620 15:01:26.615525 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.615573 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260620 15:01:26.792307 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.791955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:49581)
0620 15:01:26.792162 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.792211 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":58,"threads_started":1}
I20260620 15:01:27.214507 24309 raft_consensus.cc:482] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:27.214746 24309 raft_consensus.cc:504] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:27.216060 24309 leader_election.cc:288] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213), 9186149806774e538dfd30894999724f (127.7.46.131:42993)
Jun 20 15:01:27 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.130@KRBTEST.COM
Jun 20 15:01:27 dist-test-slave-30t4 krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.131@KRBTEST.COM
I20260620 15:01:27.230123 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:27.216241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:34213 (local address 127.7.46.129:35659)
0620 15:01:27.216462 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:27.216495 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:27.216621 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:27.218184 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:27.218190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:27.218751 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:27.219233 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:27.219247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:27.220563 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:27.220567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:27.221449 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:27.221459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:27.222145 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:27.222149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:27.222400 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:27.223041 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:27.223056 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:27.224728 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:27.227358 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:27.227364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:27.227383 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:27.227670 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:27.228060 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:27.228062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:27.228064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:27.228188 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:27.228970 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:27.228976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:27.229331 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 15:01:27.229554 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 15:01:27.229913 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":262,"spinlock_wait_cycles":13696,"thread_start_us":75,"threads_started":1}
I20260620 15:01:27.230161 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:27.216853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:38729 (local address 127.7.46.131:42993)
0620 15:01:27.217133 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:27.217152 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:27.217183 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:01:27.217786 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:27.217791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:27.218079 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:27.218353 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:27.219404 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:27.220419 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:27.221566 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:27.222010 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:27.222195 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:27.225282 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:27.225287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:27.225298 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:27.225325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:27.227573 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:27.228019 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:27.228027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:27.228035 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:27.228157 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:27.228448 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:27.228451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:27.228452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:27.228845 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:27.228979 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:27.229699 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 15:01:27.229976 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260620 15:01:27.230185 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:27.216382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:35659 (local address 127.7.46.130:34213)
0620 15:01:27.216672 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:27.216691 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:27.216739 (+ 48us) server_negotiation.cc:407] Connection header received
0620 15:01:27.217480 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:27.217487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:27.217826 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:27.218153 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:27.219406 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:27.220423 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:27.221575 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:27.222013 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:27.222195 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:27.224956 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:27.224962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:27.224973 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:27.224998 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:27.227214 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:27.227822 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:27.227828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:27.227834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:27.227950 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:27.228329 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:27.228334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:27.228337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:27.228845 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:27.228979 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:27.229703 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 15:01:27.229976 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260620 15:01:27.230123 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:27.216581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:42993 (local address 127.7.46.129:38729)
0620 15:01:27.216845 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:27.216858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:27.216943 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:27.218295 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:27.218298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:27.218767 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:27.219232 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:27.219246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:27.220572 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:27.220575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:27.221446 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:27.221454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:27.222136 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:27.222140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:27.222398 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:27.223096 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:27.223108 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:27.225094 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:27.227671 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:27.227674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:27.227676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:27.227919 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:27.228248 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:27.228253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:27.228254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:27.228368 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:27.228970 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:27.228976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:27.229331 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 15:01:27.229554 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 15:01:27.229963 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":341,"spinlock_wait_cycles":24960,"thread_start_us":61,"threads_started":1}
W20260620 15:01:27.232810 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9186149806774e538dfd30894999724f (127.7.46.131:42993): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
W20260620 15:01:27.233032 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
I20260620 15:01:27.233105 23923 leader_election.cc:302] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 378e968466b446ff8221671810afd5eb; no voters: 9186149806774e538dfd30894999724f, f3d61096013345c38e38c07bbeac36bc
I20260620 15:01:27.233601 24309 raft_consensus.cc:2726] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:01:27.462731 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:27.462462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:57121)
0620 15:01:27.462594 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:27.462649 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 15:01:27.616842 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:27.616444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:33929)
0620 15:01:27.616691 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:27.616743 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260620 15:01:27.793290 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:27.792927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:38659)
0620 15:01:27.793154 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:27.793204 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260620 15:01:28.463953 24316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:28.463486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:48841)
0620 15:01:28.463742 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:28.463857 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260620 15:01:28.618022 24317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:28.617560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:36343)
0620 15:01:28.617835 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:28.617887 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260620 15:01:28.794306 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:28.793937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:37955)
0620 15:01:28.794155 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:28.794215 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260620 15:01:29.217314 24319 raft_consensus.cc:482] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:29.217454 24319 raft_consensus.cc:504] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:29.217811 24319 leader_election.cc:288] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213), 9186149806774e538dfd30894999724f (127.7.46.131:42993)
W20260620 15:01:29.218657 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
W20260620 15:01:29.218852 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9186149806774e538dfd30894999724f (127.7.46.131:42993): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
I20260620 15:01:29.218991 23923 leader_election.cc:302] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 378e968466b446ff8221671810afd5eb; no voters: 9186149806774e538dfd30894999724f, f3d61096013345c38e38c07bbeac36bc
I20260620 15:01:29.219142 24319 raft_consensus.cc:2726] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:01:29.465036 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:29.464654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:56223)
0620 15:01:29.464892 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:29.464943 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260620 15:01:29.619163 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:29.618745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:36559)
0620 15:01:29.618987 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:29.619043 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260620 15:01:29.795370 24322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:29.794985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:45831)
0620 15:01:29.795236 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:29.795288 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260620 15:01:30.466516 24323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:30.466054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:35631)
0620 15:01:30.466365 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:30.466422 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
I20260620 15:01:30.620293 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:30.619911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:49087)
0620 15:01:30.620142 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:30.620198 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260620 15:01:30.797212 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:30.796129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:46955)
0620 15:01:30.796984 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:30.797061 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":89,"threads_started":1}
I20260620 15:01:31.467721 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:31.467254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:54229)
0620 15:01:31.467561 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:31.467613 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":84,"threads_started":1}
I20260620 15:01:31.621464 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:31.621059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:47833)
0620 15:01:31.621299 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:31.621351 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260620 15:01:31.801000 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:31.800654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:56855)
0620 15:01:31.800868 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:31.800917 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260620 15:01:32.161913 24329 raft_consensus.cc:482] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:32.162055 24329 raft_consensus.cc:504] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:32.162463 24329 leader_election.cc:288] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213), 9186149806774e538dfd30894999724f (127.7.46.131:42993)
W20260620 15:01:32.163262 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
W20260620 15:01:32.163427 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9186149806774e538dfd30894999724f (127.7.46.131:42993): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
I20260620 15:01:32.163491 23923 leader_election.cc:302] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 378e968466b446ff8221671810afd5eb; no voters: 9186149806774e538dfd30894999724f, f3d61096013345c38e38c07bbeac36bc
I20260620 15:01:32.163636 24329 raft_consensus.cc:2726] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:01:32.468945 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:32.468516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:47743)
0620 15:01:32.468773 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:32.468829 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
I20260620 15:01:32.622666 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:32.622204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:39115)
0620 15:01:32.622492 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:32.622556 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
I20260620 15:01:32.802093 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:32.801671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:33735)
0620 15:01:32.801940 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:32.801992 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":67,"threads_started":1}
I20260620 15:01:33.470165 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:33.469783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:42711)
0620 15:01:33.470013 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:33.470065 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260620 15:01:33.623890 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:33.623405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:58265)
0620 15:01:33.623668 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:33.623726 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260620 15:01:33.803148 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:33.802796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:56409)
0620 15:01:33.803017 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:33.803064 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260620 15:01:34.471408 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:34.471012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:36523)
0620 15:01:34.471268 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:34.471319 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260620 15:01:34.514397 24337 raft_consensus.cc:482] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:34.514559 24337 raft_consensus.cc:504] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:34.514974 24337 leader_election.cc:288] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213), 9186149806774e538dfd30894999724f (127.7.46.131:42993)
W20260620 15:01:34.515857 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
W20260620 15:01:34.515995 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9186149806774e538dfd30894999724f (127.7.46.131:42993): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
I20260620 15:01:34.516049 23923 leader_election.cc:302] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 378e968466b446ff8221671810afd5eb; no voters: 9186149806774e538dfd30894999724f, f3d61096013345c38e38c07bbeac36bc
I20260620 15:01:34.516223 24337 raft_consensus.cc:2726] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:01:34.624897 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:34.624538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:50897)
0620 15:01:34.624757 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:34.624809 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20260620 15:01:34.804291 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:34.803889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:41265)
0620 15:01:34.804120 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:34.804173 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
I20260620 15:01:35.472564 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:35.472194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:49227)
0620 15:01:35.472417 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:35.472466 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260620 15:01:35.625942 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:35.625571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:32821)
0620 15:01:35.625800 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:35.625851 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260620 15:01:35.805470 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:35.805084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:47741)
0620 15:01:35.805327 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:35.805381 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260620 15:01:36.473654 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:36.473241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:49343)
0620 15:01:36.473500 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:36.473552 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260620 15:01:36.627178 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:36.626791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:46885)
0620 15:01:36.627028 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:36.627080 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260620 15:01:36.806615 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:36.806259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:60355)
0620 15:01:36.806478 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:36.806527 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260620 15:01:37.474779 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:37.474326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:53313)
0620 15:01:37.474629 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:37.474685 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":70,"threads_started":1}
I20260620 15:01:37.628252 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:37.627889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:38451)
0620 15:01:37.628102 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:37.628159 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260620 15:01:37.807760 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:37.807360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:57147)
0620 15:01:37.807621 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:37.807673 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260620 15:01:38.322561 24349 raft_consensus.cc:482] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:38.322691 24349 raft_consensus.cc:504] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d61096013345c38e38c07bbeac36bc" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 34213 } } peers { permanent_uuid: "9186149806774e538dfd30894999724f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 42993 } } peers { permanent_uuid: "378e968466b446ff8221671810afd5eb" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 36425 } }
I20260620 15:01:38.323104 24349 leader_election.cc:288] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213), 9186149806774e538dfd30894999724f (127.7.46.131:42993)
W20260620 15:01:38.323901 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d61096013345c38e38c07bbeac36bc (127.7.46.130:34213): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
W20260620 15:01:38.324079 23923 leader_election.cc:341] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9186149806774e538dfd30894999724f (127.7.46.131:42993): Not found: Tablet not found: 9d0376f19d8d4f59815abaa78c412c7a
I20260620 15:01:38.324146 23923 leader_election.cc:302] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 378e968466b446ff8221671810afd5eb; no voters: 9186149806774e538dfd30894999724f, f3d61096013345c38e38c07bbeac36bc
I20260620 15:01:38.324342 24349 raft_consensus.cc:2726] T 9d0376f19d8d4f59815abaa78c412c7a P 378e968466b446ff8221671810afd5eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:01:38.475911 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:38.475506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:44821)
0620 15:01:38.475753 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:38.475818 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20260620 15:01:38.629377 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:38.628970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:57087)
0620 15:01:38.629219 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:38.629287 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260620 15:01:38.808944 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:38.808549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:51211)
0620 15:01:38.808791 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:38.808842 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260620 15:01:39.477084 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:39.476646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:42239)
0620 15:01:39.476921 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:39.476976 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20260620 15:01:39.630523 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:39.630136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:60313)
0620 15:01:39.630376 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:39.630429 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
I20260620 15:01:39.810091 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:39.809681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.131:39391)
0620 15:01:39.809933 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:39.809999 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260620 15:01:40.478488 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:40.478022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.129:36443)
0620 15:01:40.478289 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:40.478342 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260620 15:01:40.631678 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:40.631309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43303 (local address 127.7.46.130:50129)
0620 15:01:40.631538 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:40.631588 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.190:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
I20260620 15:01:40.799933 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 23866
I20260620 15:01:40.805697 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 24038
I20260620 15:01:40.810979 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 24173
2026-06-20T15:01:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59464 ms)
[----------] 30 tests from SecurityITest (236384 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:40 dist-test-slave-30t4 krb5kdc[24361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:40 dist-test-slave-30t4 krb5kdc[24361](info): set up 2 sockets
Jun 20 15:01:40 dist-test-slave-30t4 krb5kdc[24361](info): commencing operation
krb5kdc: starting...
W20260620 15:01:41.851557 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:41 dist-test-slave-30t4 krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:01:41.964396 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:36371
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:36371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:42.070518 24374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:42.070838 24374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:42.070931 24374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:42.074929 24374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:42.075006 24374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:42.075052 24374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:42.075083 24374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:42.075112 24374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:42.080248 24374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:36371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:36371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:42.081151 24374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:42.082067 24374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:42.090308 24379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:42.090355 24374 server_base.cc:1034] running on GCE node
W20260620 15:01:42.090306 24380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:42.090306 24382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:42.091009 24374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:42.092083 24374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:42.093262 24374 hybrid_clock.cc:648] HybridClock initialized: now 1781967702093242 us; error 39 us; skew 500 ppm
Jun 20 15:01:42 dist-test-slave-30t4 krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:42.095940 24374 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:01:42.096701 24374 webserver.cc:460] Webserver started at http://127.7.46.190:35401/ using document root <none> and password file <none>
I20260620 15:01:42.097204 24374 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:42.097290 24374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:42.097501 24374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:42.099334 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "55d57068fd3c478fa2aea88b6b5ad3cf"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "0c826c76af9d16ed41cf36e10657fa46"
server_key_iv: "3975b1d0e52fb17df2f28457fb6e40d5"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.099937 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "55d57068fd3c478fa2aea88b6b5ad3cf"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "0c826c76af9d16ed41cf36e10657fa46"
server_key_iv: "3975b1d0e52fb17df2f28457fb6e40d5"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.103292 24374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:42.105544 24389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.106628 24374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:01:42.106765 24374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "55d57068fd3c478fa2aea88b6b5ad3cf"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "0c826c76af9d16ed41cf36e10657fa46"
server_key_iv: "3975b1d0e52fb17df2f28457fb6e40d5"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.106869 24374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:42.121105 24374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:42.124101 24374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:42.124289 24374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:42.130656 24374 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:36371
I20260620 15:01:42.130721 24441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:36371 every 8 connection(s)
I20260620 15:01:42.131515 24374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:42.134131 24442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:42.139178 24442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf: Bootstrap starting.
I20260620 15:01:42.139686 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 24374
I20260620 15:01:42.139829 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:42.140146 7354 external_mini_cluster.cc:1428] Setting key 26a8465c85b73cc76be51ccb2c7dd06c
I20260620 15:01:42.141923 24442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:42.142611 24442 log.cc:824] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:42.144659 24442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf: No bootstrap required, opened a new log
Jun 20 15:01:42 dist-test-slave-30t4 krb5kdc[24361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:42.148927 24442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d57068fd3c478fa2aea88b6b5ad3cf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36371 } }
I20260620 15:01:42.149202 24442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:42.149281 24442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d57068fd3c478fa2aea88b6b5ad3cf, State: Initialized, Role: FOLLOWER
I20260620 15:01:42.149757 24442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d57068fd3c478fa2aea88b6b5ad3cf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36371 } }
I20260620 15:01:42.149926 24442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:42.150002 24442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:42.150100 24442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:42.151211 24442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d57068fd3c478fa2aea88b6b5ad3cf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36371 } }
I20260620 15:01:42.151568 24442 leader_election.cc:302] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55d57068fd3c478fa2aea88b6b5ad3cf; no voters:
I20260620 15:01:42.151917 24442 leader_election.cc:288] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:42.152019 24447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:42.152230 24447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [term 1 LEADER]: Becoming Leader. State: Replica: 55d57068fd3c478fa2aea88b6b5ad3cf, State: Running, Role: LEADER
I20260620 15:01:42.152554 24447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d57068fd3c478fa2aea88b6b5ad3cf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36371 } }
I20260620 15:01:42.153115 24442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:42.154492 24449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55d57068fd3c478fa2aea88b6b5ad3cf. Latest consensus state: current_term: 1 leader_uuid: "55d57068fd3c478fa2aea88b6b5ad3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d57068fd3c478fa2aea88b6b5ad3cf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36371 } } }
I20260620 15:01:42.154493 24448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55d57068fd3c478fa2aea88b6b5ad3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d57068fd3c478fa2aea88b6b5ad3cf" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36371 } } }
I20260620 15:01:42.154633 24449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:42.154652 24448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:42.155164 24456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:42.155231 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.141418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.7.46.190:36371)
0620 15:01:42.141826 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.141840 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.141874 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:01:42.142513 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.142530 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.142882 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.143256 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:42.144109 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.145108 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.146280 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.146731 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.146845 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.149624 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:42.149630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:42.149644 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:42.149672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:42.151657 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.152266 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.152272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.152278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.152414 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.152833 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.152837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.152839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.153319 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:42.153518 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.153830 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.154154 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260620 15:01:42.158279 24456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:42.163552 24456 catalog_manager.cc:1347] Generated new cluster ID: 57770b126e404c4195b318e16f02bf74
I20260620 15:01:42.163611 24456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:42.172636 24456 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:42.173748 24456 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:42.184672 24456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf: Generated new TSK 0
I20260620 15:01:42.185415 24456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 15:01:42.218163 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36371
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:42.324671 24468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:42.324892 24468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:42.324932 24468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:42.328824 24468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:42.328871 24468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:42.328927 24468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:01:42.333899 24468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:42.334749 24468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:42.335633 24468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:42.344874 24476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:42.344856 24473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:42.344905 24468 server_base.cc:1034] running on GCE node
W20260620 15:01:42.344841 24474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:42.345391 24468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:42.345990 24468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:42.347196 24468 hybrid_clock.cc:648] HybridClock initialized: now 1781967702347182 us; error 24 us; skew 500 ppm
Jun 20 15:01:42 dist-test-slave-30t4 krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:42.349790 24468 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:01:42.350524 24468 webserver.cc:460] Webserver started at http://127.7.46.129:40015/ using document root <none> and password file <none>
I20260620 15:01:42.350999 24468 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:42.351063 24468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:42.351212 24468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:42.353179 24468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "f792fd4db9ff40229131b9954fd67664"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "d1766467f39d84150b5b8aa441450d65"
server_key_iv: "ffba87e2ebe9e38d974d4ab4bb057e02"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.353745 24468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "f792fd4db9ff40229131b9954fd67664"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "d1766467f39d84150b5b8aa441450d65"
server_key_iv: "ffba87e2ebe9e38d974d4ab4bb057e02"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.357366 24468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:01:42.359651 24483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.360869 24468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:42.361019 24468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "f792fd4db9ff40229131b9954fd67664"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "d1766467f39d84150b5b8aa441450d65"
server_key_iv: "ffba87e2ebe9e38d974d4ab4bb057e02"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.361119 24468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:42.373234 24468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:42.376047 24468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:42.376233 24468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:42.376746 24468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:42.377485 24468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:42.377557 24468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.377625 24468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:42.377667 24468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.386049 24468 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:45877
I20260620 15:01:42.386114 24596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:45877 every 8 connection(s)
I20260620 15:01:42.386875 24468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:42.393594 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 24468
I20260620 15:01:42.393743 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:42.394023 7354 external_mini_cluster.cc:1428] Setting key fb5c4e4dd9b7ae3f2171a08e6b6f274f
Jun 20 15:01:42 dist-test-slave-30t4 krb5kdc[24361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:42.400089 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.388579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:59463 (local address 127.7.46.190:36371)
0620 15:01:42.388707 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.388711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.389475 (+ 764us) server_negotiation.cc:407] Connection header received
0620 15:01:42.390293 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.390296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.390352 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.390432 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:42.391464 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.392042 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.393124 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.393365 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.393499 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.396239 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:42.396242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:42.396245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:42.396270 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:42.397979 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.398583 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.398586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.398587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.398657 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.399022 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.399024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.399025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.399180 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:42.399254 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.399883 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.399977 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:01:42.401228 24599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.388920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36371 (local address 127.7.46.129:59463)
0620 15:01:42.389327 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:42.389365 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:42.390091 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:42.390589 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:42.390595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:42.390903 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:42.391322 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.391333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.392148 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.392150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.393007 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.393015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.393537 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.393541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.393741 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.394417 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:42.394432 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:42.396075 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:42.398103 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:42.398108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:42.398139 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:42.398488 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:42.398745 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:42.398751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:42.398756 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:42.398932 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:42.399283 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:42.399292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:42.399591 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 15:01:42.399843 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 15:01:42.400185 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260620 15:01:42.402460 24597 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36371
I20260620 15:01:42.402685 24597 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:42.403146 24597 heartbeater.cc:511] Master 127.7.46.190:36371 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 15:01:42.404626 24406 ts_manager.cc:194] Registered new tserver with Master: f792fd4db9ff40229131b9954fd67664 (127.7.46.129:45877)
I20260620 15:01:42.405856 24406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:59463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 15:01:42.423228 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36371
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:42.529659 24602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:42.529929 24602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:42.530006 24602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:42.534162 24602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:42.534274 24602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:42.534368 24602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:01:42.539500 24602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:42.540560 24602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:42.541667 24602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:42.551018 24608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:42.550966 24610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:42.550985 24607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:42.551276 24602 server_base.cc:1034] running on GCE node
I20260620 15:01:42.551685 24602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:42.552356 24602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:42.553530 24602 hybrid_clock.cc:648] HybridClock initialized: now 1781967702553495 us; error 46 us; skew 500 ppm
Jun 20 15:01:42 dist-test-slave-30t4 krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:42.556392 24602 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:01:42.557140 24602 webserver.cc:460] Webserver started at http://127.7.46.130:45951/ using document root <none> and password file <none>
I20260620 15:01:42.557627 24602 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:42.557682 24602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:42.557827 24602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:42.559689 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "580cfdf2a04246efa9e7fd6a11b70a32"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "fc7434abf3d39bba2dc906601f92afcd"
server_key_iv: "f493fb4883ba89e743293abac60840cd"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.560236 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "580cfdf2a04246efa9e7fd6a11b70a32"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "fc7434abf3d39bba2dc906601f92afcd"
server_key_iv: "f493fb4883ba89e743293abac60840cd"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.563547 24602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:42.565976 24617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.567076 24602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:42.567173 24602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "580cfdf2a04246efa9e7fd6a11b70a32"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "fc7434abf3d39bba2dc906601f92afcd"
server_key_iv: "f493fb4883ba89e743293abac60840cd"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.567247 24602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:42.582013 24602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:42.584563 24602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:42.584697 24602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:42.585150 24602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:42.585857 24602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:42.585901 24602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.585940 24602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:42.585961 24602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.594735 24602 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:33781
I20260620 15:01:42.594774 24730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:33781 every 8 connection(s)
I20260620 15:01:42.595556 24602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:42.598981 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 24602
I20260620 15:01:42.599124 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:42.599474 7354 external_mini_cluster.cc:1428] Setting key d65e1e81d9f9b19007e32c4a35b885e7
Jun 20 15:01:42 dist-test-slave-30t4 krb5kdc[24361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
I20260620 15:01:42.610121 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.597254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:33485 (local address 127.7.46.190:36371)
0620 15:01:42.597414 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.597419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.598201 (+ 782us) server_negotiation.cc:407] Connection header received
0620 15:01:42.599224 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.599229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.599292 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.599353 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:42.601054 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.601587 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.603152 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.603547 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.603675 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.606226 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:42.606235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:42.606237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:42.606262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:42.608027 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.608555 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.608559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.608561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.608643 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.608988 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.608992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.608994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.609183 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:42.609297 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.609819 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.609936 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:01:42.611079 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.597609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36371 (local address 127.7.46.130:33485)
0620 15:01:42.598046 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:42.598094 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:42.598948 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:42.599751 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:42.599760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:42.600282 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:42.600844 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.600870 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.601723 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.601729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.603023 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.603033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.603694 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.603697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.603902 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.604490 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:42.604506 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:42.605914 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:42.608153 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:42.608159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:42.608173 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:42.608441 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:42.608744 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:42.608748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:42.608750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:42.608887 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:42.609294 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:42.609301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:42.609577 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 15:01:42.609767 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 15:01:42.610133 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
I20260620 15:01:42.612941 24731 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36371
I20260620 15:01:42.613205 24731 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:42.613772 24731 heartbeater.cc:511] Master 127.7.46.190:36371 requested a full tablet report, sending...
I20260620 15:01:42.615011 24406 ts_manager.cc:194] Registered new tserver with Master: 580cfdf2a04246efa9e7fd6a11b70a32 (127.7.46.130:33781)
I20260620 15:01:42.615813 24406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:33485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 15:01:42.628930 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36371
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:42.735349 24736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:42.735567 24736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:42.735638 24736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:42.739496 24736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:42.739547 24736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:42.739600 24736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:01:42.744648 24736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:36371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:42.745487 24736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:42.746407 24736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:42.757543 24744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:42.757550 24741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:42.757699 24736 server_base.cc:1034] running on GCE node
W20260620 15:01:42.757552 24742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:42.758148 24736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:42.758666 24736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:42.759835 24736 hybrid_clock.cc:648] HybridClock initialized: now 1781967702759821 us; error 29 us; skew 500 ppm
Jun 20 15:01:42 dist-test-slave-30t4 krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:42.762413 24736 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:01:42.763126 24736 webserver.cc:460] Webserver started at http://127.7.46.131:41897/ using document root <none> and password file <none>
I20260620 15:01:42.763608 24736 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:42.763662 24736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:42.763887 24736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:42.766039 24736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "4ed0500ec03448efa3245fb4a400bbbb"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "2b6b90d8946fb1dfc03cd53156795e0b"
server_key_iv: "46270825aa8c58d7671e63aca45dd724"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.766525 24736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "4ed0500ec03448efa3245fb4a400bbbb"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "2b6b90d8946fb1dfc03cd53156795e0b"
server_key_iv: "46270825aa8c58d7671e63aca45dd724"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.769920 24736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:01:42.772154 24751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.773288 24736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:42.773402 24736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "4ed0500ec03448efa3245fb4a400bbbb"
format_stamp: "Formatted at 2026-06-20 15:01:42 on dist-test-slave-30t4"
server_key: "2b6b90d8946fb1dfc03cd53156795e0b"
server_key_iv: "46270825aa8c58d7671e63aca45dd724"
server_key_version: "encryptionkey@0"
I20260620 15:01:42.773479 24736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:42.792351 24736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:42.794951 24736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:42.795084 24736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:42.795538 24736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:42.796320 24736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:42.796373 24736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.796409 24736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:42.796430 24736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:42.804711 24736 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:40351
I20260620 15:01:42.804736 24864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:40351 every 8 connection(s)
I20260620 15:01:42.805536 24736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
Jun 20 15:01:42 dist-test-slave-30t4 krb5kdc[24361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967702, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:42.814893 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 24736
I20260620 15:01:42.815040 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:42.815400 7354 external_mini_cluster.cc:1428] Setting key 0141baf2be459bf5ea16ff1b7c537421
I20260620 15:01:42.818552 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.807259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:37703 (local address 127.7.46.190:36371)
0620 15:01:42.807389 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.807393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.808158 (+ 765us) server_negotiation.cc:407] Connection header received
0620 15:01:42.809036 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.809040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.809100 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.809201 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:42.810323 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.810853 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.812017 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.812235 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.812341 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.814655 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:42.814659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:42.814662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:42.814691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:42.816385 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.816940 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.816942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.816943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.817014 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.817385 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.817391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.817394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.817628 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:42.817713 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.818262 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.818377 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 15:01:42.819412 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.807544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:36371 (local address 127.7.46.131:37703)
0620 15:01:42.808018 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:42.808058 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:42.808823 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:42.809367 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:42.809393 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:42.809735 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:42.810178 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.810190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.810977 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.810981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.811896 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.811905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.812385 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.812389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.812551 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.813049 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:42.813063 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:42.814487 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:42.816479 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:42.816483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:42.816493 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:42.816849 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:42.817102 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:42.817105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:42.817107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:42.817268 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:42.817730 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:42.817735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:42.817940 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 15:01:42.818190 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 15:01:42.818484 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":105,"threads_started":1}
I20260620 15:01:42.820405 24865 heartbeater.cc:347] Connected to a master server at 127.7.46.190:36371
I20260620 15:01:42.820631 24865 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:42.821082 24865 heartbeater.cc:511] Master 127.7.46.190:36371 requested a full tablet report, sending...
I20260620 15:01:42.822026 24406 ts_manager.cc:194] Registered new tserver with Master: 4ed0500ec03448efa3245fb4a400bbbb (127.7.46.131:40351)
I20260620 15:01:42.822610 24406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:37703
I20260620 15:01:42.829010 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:01:42.837574 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.830951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.7.46.190:36371)
0620 15:01:42.831133 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.831136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.831152 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:01:42.831257 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.831259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.831292 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.831367 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:42.832091 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.832592 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.833386 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.833597 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.833675 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.834894 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:42.834896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:42.834898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:42.834918 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:42.836179 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.836539 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.836542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.836543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.836616 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:42.836839 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:42.836841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:42.836842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:42.837025 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:42.837119 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.837293 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.837371 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 15:01:42.840154 24406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:01:42.842998 24406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:01:42.856984 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.850764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.7.46.129:45877)
0620 15:01:42.851111 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.851124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.851573 (+ 449us) server_negotiation.cc:407] Connection header received
0620 15:01:42.851860 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.851865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.852088 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.852251 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:42.852572 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.853720 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.855490 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.856310 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.856421 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.856559 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.856733 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.856840 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":140,"threads_started":1}
I20260620 15:01:42.856983 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.850627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45877 (local address 127.0.0.1:50648)
0620 15:01:42.851296 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:42.851475 (+ 179us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:42.851640 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:42.852257 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:42.852263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:42.852286 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:42.852420 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.852427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.853881 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.853884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.855358 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.855368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.856434 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.856438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.856544 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.856586 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 15:01:42.856702 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 15:01:42.856824 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":157,"thread_start_us":65,"threads_started":1}
I20260620 15:01:42.857245 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.850758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:40351 (local address 127.0.0.1:53194)
0620 15:01:42.851509 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:42.851523 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:42.851640 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:42.853173 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:42.853176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:42.853192 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:42.853266 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.853271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.854674 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.854677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.855725 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.855733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.856830 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.856834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.856921 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.856942 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:01:42.857015 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 15:01:42.857099 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260620 15:01:42.857295 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.851343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.7.46.131:40351)
0620 15:01:42.852624 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.852637 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.852657 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:01:42.852722 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.852727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.852892 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.853023 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:42.853384 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.854530 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.856030 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.856721 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.856816 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.856936 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.857053 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.857146 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":139,"threads_started":1}
I20260620 15:01:42.857687 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.851283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:33781 (local address 127.0.0.1:47714)
0620 15:01:42.851577 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:42.851591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:42.851676 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:42.852497 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:42.852500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:42.852518 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:42.852630 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.852639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.853833 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.853840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.855284 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.855293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.857304 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.857311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.857405 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.857434 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:01:42.857509 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 15:01:42.857572 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":96,"thread_start_us":87,"threads_started":1}
I20260620 15:01:42.857777 24881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.851728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.7.46.130:33781)
0620 15:01:42.852089 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.852101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.852122 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:01:42.852196 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.852203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.852362 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.852506 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:42.852759 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.853700 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.855460 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.856310 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.856421 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.856564 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.857542 (+ 978us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.857634 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":146,"threads_started":1}
I20260620 15:01:42.860226 24799 tablet_service.cc:1467] Processing CreateTablet for tablet d3edb5ac481c4477a3b583ba7fb953bc (DEFAULT_TABLE table=test-table [id=d2a3d7813ed14ee2a24deb32205a0a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:42.860224 24531 tablet_service.cc:1467] Processing CreateTablet for tablet d3edb5ac481c4477a3b583ba7fb953bc (DEFAULT_TABLE table=test-table [id=d2a3d7813ed14ee2a24deb32205a0a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:42.860430 24665 tablet_service.cc:1467] Processing CreateTablet for tablet d3edb5ac481c4477a3b583ba7fb953bc (DEFAULT_TABLE table=test-table [id=d2a3d7813ed14ee2a24deb32205a0a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:42.861256 24665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3edb5ac481c4477a3b583ba7fb953bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:42.861276 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3edb5ac481c4477a3b583ba7fb953bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:42.861284 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3edb5ac481c4477a3b583ba7fb953bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:42.867585 24882 tablet_bootstrap.cc:492] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: Bootstrap starting.
I20260620 15:01:42.868993 24884 tablet_bootstrap.cc:492] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: Bootstrap starting.
I20260620 15:01:42.868993 24883 tablet_bootstrap.cc:492] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664: Bootstrap starting.
I20260620 15:01:42.870085 24882 tablet_bootstrap.cc:654] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:42.871168 24884 tablet_bootstrap.cc:654] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:42.871168 24883 tablet_bootstrap.cc:654] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:42.871197 24882 log.cc:824] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:42.872471 24884 log.cc:824] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:42.872471 24883 log.cc:824] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:42.873261 24882 tablet_bootstrap.cc:492] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: No bootstrap required, opened a new log
I20260620 15:01:42.873454 24882 ts_tablet_manager.cc:1397] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 15:01:42.874343 24883 tablet_bootstrap.cc:492] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664: No bootstrap required, opened a new log
I20260620 15:01:42.874518 24884 tablet_bootstrap.cc:492] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: No bootstrap required, opened a new log
I20260620 15:01:42.874534 24883 ts_tablet_manager.cc:1397] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 15:01:42.874711 24884 ts_tablet_manager.cc:1397] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 15:01:42.877282 24882 raft_consensus.cc:348] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.877549 24882 raft_consensus.cc:374] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:42.877612 24882 raft_consensus.cc:729] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cfdf2a04246efa9e7fd6a11b70a32, State: Initialized, Role: FOLLOWER
I20260620 15:01:42.877763 24884 raft_consensus.cc:348] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.878059 24884 raft_consensus.cc:374] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:42.878126 24884 raft_consensus.cc:729] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed0500ec03448efa3245fb4a400bbbb, State: Initialized, Role: FOLLOWER
I20260620 15:01:42.878065 24882 consensus_queue.cc:260] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.878448 24883 raft_consensus.cc:348] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.878671 24883 raft_consensus.cc:374] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:42.878698 24731 heartbeater.cc:503] Master 127.7.46.190:36371 was elected leader, sending a full tablet report...
I20260620 15:01:42.878618 24884 consensus_queue.cc:260] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.878732 24883 raft_consensus.cc:729] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f792fd4db9ff40229131b9954fd67664, State: Initialized, Role: FOLLOWER
I20260620 15:01:42.879256 24865 heartbeater.cc:503] Master 127.7.46.190:36371 was elected leader, sending a full tablet report...
I20260620 15:01:42.879628 24884 ts_tablet_manager.cc:1428] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 15:01:42.879802 24883 consensus_queue.cc:260] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.880687 24883 ts_tablet_manager.cc:1428] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 15:01:42.880831 24597 heartbeater.cc:503] Master 127.7.46.190:36371 was elected leader, sending a full tablet report...
I20260620 15:01:42.882128 24882 ts_tablet_manager.cc:1428] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260620 15:01:42.887925 24598 tablet.cc:2369] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:42.940948 24888 raft_consensus.cc:482] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:42.941315 24888 raft_consensus.cc:504] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.942510 24888 leader_election.cc:288] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f792fd4db9ff40229131b9954fd67664 (127.7.46.129:45877), 4ed0500ec03448efa3245fb4a400bbbb (127.7.46.131:40351)
I20260620 15:01:42.946954 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.942971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:57043 (local address 127.7.46.131:40351)
0620 15:01:42.943126 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.943133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.943153 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:01:42.943220 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.943222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.943265 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.943377 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:42.943667 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.944250 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.945822 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.946405 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.946524 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.946575 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.946746 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.946818 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:01:42.946974 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.942773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:40351 (local address 127.7.46.130:57043)
0620 15:01:42.942942 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:42.942960 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:42.943084 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:42.943395 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:42.943398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:42.943415 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:42.943544 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.943550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.944387 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.944392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.945692 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.945704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.946530 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.946533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.946618 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.946639 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:01:42.946734 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:01:42.946813 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":1}
I20260620 15:01:42.947216 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.942902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:54031 (local address 127.7.46.129:45877)
0620 15:01:42.943064 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.943069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.943459 (+ 390us) server_negotiation.cc:407] Connection header received
0620 15:01:42.943609 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.943612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.943660 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.943742 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:42.944022 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.944675 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.946069 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.946640 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.946775 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.946818 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.947022 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.947097 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:01:42.947232 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.942773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45877 (local address 127.7.46.130:54031)
0620 15:01:42.943356 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:42.943367 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:42.943495 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:42.943753 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:42.943755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:42.943801 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:42.943916 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.943921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.944796 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.944800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.945959 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:42.945966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.946789 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.946793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:42.946887 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.946912 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:01:42.946997 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:01:42.947070 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":284,"thread_start_us":120,"threads_started":1}
I20260620 15:01:42.947641 24819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3edb5ac481c4477a3b583ba7fb953bc" candidate_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed0500ec03448efa3245fb4a400bbbb" is_pre_election: true
I20260620 15:01:42.947983 24819 raft_consensus.cc:2454] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580cfdf2a04246efa9e7fd6a11b70a32 in term 0.
I20260620 15:01:42.948038 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3edb5ac481c4477a3b583ba7fb953bc" candidate_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f792fd4db9ff40229131b9954fd67664" is_pre_election: true
I20260620 15:01:42.948339 24551 raft_consensus.cc:2454] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580cfdf2a04246efa9e7fd6a11b70a32 in term 0.
I20260620 15:01:42.948464 24620 leader_election.cc:302] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed0500ec03448efa3245fb4a400bbbb, 580cfdf2a04246efa9e7fd6a11b70a32; no voters:
I20260620 15:01:42.948748 24888 raft_consensus.cc:2781] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:01:42.948851 24888 raft_consensus.cc:482] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:01:42.948902 24888 raft_consensus.cc:3037] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:42.949952 24888 raft_consensus.cc:504] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.950307 24888 leader_election.cc:288] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [CANDIDATE]: Term 1 election: Requested vote from peers f792fd4db9ff40229131b9954fd67664 (127.7.46.129:45877), 4ed0500ec03448efa3245fb4a400bbbb (127.7.46.131:40351)
I20260620 15:01:42.950646 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3edb5ac481c4477a3b583ba7fb953bc" candidate_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f792fd4db9ff40229131b9954fd67664"
I20260620 15:01:42.950670 24819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3edb5ac481c4477a3b583ba7fb953bc" candidate_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ed0500ec03448efa3245fb4a400bbbb"
I20260620 15:01:42.950791 24819 raft_consensus.cc:3037] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:42.950765 24551 raft_consensus.cc:3037] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:42.951865 24819 raft_consensus.cc:2454] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cfdf2a04246efa9e7fd6a11b70a32 in term 1.
I20260620 15:01:42.952018 24551 raft_consensus.cc:2454] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cfdf2a04246efa9e7fd6a11b70a32 in term 1.
I20260620 15:01:42.952168 24620 leader_election.cc:302] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed0500ec03448efa3245fb4a400bbbb, 580cfdf2a04246efa9e7fd6a11b70a32; no voters:
I20260620 15:01:42.952358 24888 raft_consensus.cc:2781] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:42.952629 24888 raft_consensus.cc:686] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 1 LEADER]: Becoming Leader. State: Replica: 580cfdf2a04246efa9e7fd6a11b70a32, State: Running, Role: LEADER
I20260620 15:01:42.952972 24888 consensus_queue.cc:237] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } }
I20260620 15:01:42.956235 24405 catalog_manager.cc:5495] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 580cfdf2a04246efa9e7fd6a11b70a32 (127.7.46.130). New cstate: current_term: 1 leader_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "580cfdf2a04246efa9e7fd6a11b70a32" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 33781 } health_report { overall_health: HEALTHY } } }
I20260620 15:01:42.977283 24881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:42.973648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.7.46.130:33781)
0620 15:01:42.973772 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:42.973776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:42.973791 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:01:42.973964 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:42.973967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:42.974024 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:42.974135 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:42.974378 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.974881 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.975739 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:42.976039 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:42.976163 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:42.976460 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:42.976878 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:01:42.977007 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:42.977067 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 15:01:42.977120 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 15:01:42.982885 24819 raft_consensus.cc:1261] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 1 FOLLOWER]: Refusing update from remote peer 580cfdf2a04246efa9e7fd6a11b70a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:42.982888 24551 raft_consensus.cc:1261] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [term 1 FOLLOWER]: Refusing update from remote peer 580cfdf2a04246efa9e7fd6a11b70a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:42.983516 24888 consensus_queue.cc:1035] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f792fd4db9ff40229131b9954fd67664" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:42.983695 24892 consensus_queue.cc:1035] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed0500ec03448efa3245fb4a400bbbb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 40351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:42.990499 24895 mvcc.cc:205] Tried to move back new op lower bound from 7298939711410515968 to 7298939711297867776. Current Snapshot: MvccSnapshot[applied={T|T < 7298939711410515968}]
I20260620 15:01:42.995507 24896 mvcc.cc:205] Tried to move back new op lower bound from 7298939711410515968 to 7298939711297867776. Current Snapshot: MvccSnapshot[applied={T|T < 7298939711410515968 or (T in {7298939711410515968})}]
I20260620 15:01:42.999542 24405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:42.999703 24405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:43.001818 24405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55d57068fd3c478fa2aea88b6b5ad3cf: Sending DeleteTablet for 3 replicas of tablet d3edb5ac481c4477a3b583ba7fb953bc
I20260620 15:01:43.002535 24799 tablet_service.cc:1514] Processing DeleteTablet for tablet d3edb5ac481c4477a3b583ba7fb953bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:42 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:53194
I20260620 15:01:43.002539 24665 tablet_service.cc:1514] Processing DeleteTablet for tablet d3edb5ac481c4477a3b583ba7fb953bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:42 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:47714
I20260620 15:01:43.002624 24531 tablet_service.cc:1514] Processing DeleteTablet for tablet d3edb5ac481c4477a3b583ba7fb953bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:42 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:50648
I20260620 15:01:43.003057 24906 tablet_replica.cc:330] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664: stopping tablet replica
I20260620 15:01:43.003196 24904 tablet_replica.cc:330] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: stopping tablet replica
I20260620 15:01:43.003448 24906 raft_consensus.cc:2229] T d3edb5ac481c4477a3b583ba7fb953bc P f792fd4db9ff40229131b9954fd67664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:43.003473 24904 raft_consensus.cc:2229] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:43.003640 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 24468
I20260620 15:01:43.003742 24904 raft_consensus.cc:2258] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:43.004498 24905 tablet_replica.cc:330] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: stopping tablet replica
I20260620 15:01:43.004725 24905 raft_consensus.cc:2229] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:01:43.005069 24905 raft_consensus.cc:2258] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:43.005657 24904 ts_tablet_manager.cc:1905] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:43.006304 24905 ts_tablet_manager.cc:1905] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:43.008436 24904 ts_tablet_manager.cc:1918] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:43.008563 24904 log.cc:1196] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/d3edb5ac481c4477a3b583ba7fb953bc
I20260620 15:01:43.008916 24904 ts_tablet_manager.cc:1939] T d3edb5ac481c4477a3b583ba7fb953bc P 4ed0500ec03448efa3245fb4a400bbbb: Deleting consensus metadata
I20260620 15:01:43.008925 24905 ts_tablet_manager.cc:1918] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:43.008999 24905 log.cc:1196] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/d3edb5ac481c4477a3b583ba7fb953bc
I20260620 15:01:43.009330 24905 ts_tablet_manager.cc:1939] T d3edb5ac481c4477a3b583ba7fb953bc P 580cfdf2a04246efa9e7fd6a11b70a32: Deleting consensus metadata
I20260620 15:01:43.010169 24392 catalog_manager.cc:4841] TS 4ed0500ec03448efa3245fb4a400bbbb (127.7.46.131:40351): tablet d3edb5ac481c4477a3b583ba7fb953bc (table test-table [id=d2a3d7813ed14ee2a24deb32205a0a56]) successfully deleted
I20260620 15:01:43.010429 24391 catalog_manager.cc:4841] TS 580cfdf2a04246efa9e7fd6a11b70a32 (127.7.46.130:33781): tablet d3edb5ac481c4477a3b583ba7fb953bc (table test-table [id=d2a3d7813ed14ee2a24deb32205a0a56]) successfully deleted
W20260620 15:01:43.010838 24391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:45877 (error 108)
I20260620 15:01:43.011123 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 24602
I20260620 15:01:43.012475 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.012200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45877 (local address 127.0.0.1:50658)
0620 15:01:43.012326 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.012402 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:01:43.012662 24391 catalog_manager.cc:4584] TS f792fd4db9ff40229131b9954fd67664 (127.7.46.129:45877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3edb5ac481c4477a3b583ba7fb953bc: Network error: Client connection negotiation failed: client connection to 127.7.46.129:45877: connect: Connection refused (error 111)
I20260620 15:01:43.016552 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 24736
I20260620 15:01:43.021929 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 24374
2026-06-20T15:01:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2210 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T15:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:43Z Disabled control of system clock
I20260620 15:01:43.051047 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43307
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:01:43.155259 24911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:43.155471 24911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:43.155503 24911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:43.159319 24911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:43.159364 24911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:43.159379 24911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:43.159396 24911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:43.159410 24911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:43.163883 24911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:43.164734 24911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:43.166064 24911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:43.174065 24917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:43.174105 24919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:43.174232 24911 server_base.cc:1034] running on GCE node
W20260620 15:01:43.174105 24916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:43.174870 24911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:43.175910 24911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:43.177084 24911 hybrid_clock.cc:648] HybridClock initialized: now 1781967703177051 us; error 46 us; skew 500 ppm
I20260620 15:01:43.178763 24911 webserver.cc:460] Webserver started at http://127.7.46.190:36925/ using document root <none> and password file <none>
I20260620 15:01:43.179289 24911 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:43.179374 24911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:43.179586 24911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:43.181507 24911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "d8dc12d34f6346e6b52ffe55952c1e65"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "d33d8885eac0cac8cd71f4d426b02b78"
server_key_iv: "23e67526badb2f5777dcc95cdc6cf68a"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.182236 24911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "d8dc12d34f6346e6b52ffe55952c1e65"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "d33d8885eac0cac8cd71f4d426b02b78"
server_key_iv: "23e67526badb2f5777dcc95cdc6cf68a"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.185829 24911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:01:43.188133 24925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.189270 24911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:43.189411 24911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "d8dc12d34f6346e6b52ffe55952c1e65"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "d33d8885eac0cac8cd71f4d426b02b78"
server_key_iv: "23e67526badb2f5777dcc95cdc6cf68a"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.189517 24911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:43.199659 24911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:43.200266 24911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:43.200454 24911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:43.207202 24911 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:43307
I20260620 15:01:43.207208 24977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:43307 every 8 connection(s)
I20260620 15:01:43.208063 24911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:43.210836 24978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:43.215948 24978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65: Bootstrap starting.
I20260620 15:01:43.216022 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 24911
I20260620 15:01:43.216140 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:43.216434 7354 external_mini_cluster.cc:1428] Setting key f917a2afc0eae0e2e75bdefe0c9a0152
I20260620 15:01:43.218331 24978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:43.219024 24978 log.cc:824] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:43.220894 24978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65: No bootstrap required, opened a new log
I20260620 15:01:43.225076 24978 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8dc12d34f6346e6b52ffe55952c1e65" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43307 } }
I20260620 15:01:43.225298 24978 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:43.225386 24978 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8dc12d34f6346e6b52ffe55952c1e65, State: Initialized, Role: FOLLOWER
I20260620 15:01:43.225646 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.217722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.7.46.190:43307)
0620 15:01:43.218298 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.218311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.218343 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:01:43.218975 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.218991 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.219447 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.219854 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:43.220168 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.221133 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.222325 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.222778 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.222904 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.223740 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:43.223745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:43.223757 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:43.224206 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:43.224259 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:43.224280 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:43.224356 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.224558 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.224842 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
I20260620 15:01:43.225848 24978 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8dc12d34f6346e6b52ffe55952c1e65" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43307 } }
I20260620 15:01:43.226002 24978 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:43.226064 24978 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:43.226156 24978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:43.227048 24978 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8dc12d34f6346e6b52ffe55952c1e65" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43307 } }
I20260620 15:01:43.227378 24978 leader_election.cc:302] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8dc12d34f6346e6b52ffe55952c1e65; no voters:
I20260620 15:01:43.227677 24978 leader_election.cc:288] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:43.227756 24983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:43.227991 24983 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [term 1 LEADER]: Becoming Leader. State: Replica: d8dc12d34f6346e6b52ffe55952c1e65, State: Running, Role: LEADER
I20260620 15:01:43.228286 24983 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8dc12d34f6346e6b52ffe55952c1e65" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43307 } }
I20260620 15:01:43.228754 24978 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:43.229854 24985 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8dc12d34f6346e6b52ffe55952c1e65. Latest consensus state: current_term: 1 leader_uuid: "d8dc12d34f6346e6b52ffe55952c1e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8dc12d34f6346e6b52ffe55952c1e65" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43307 } } }
I20260620 15:01:43.229981 24985 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:43.229856 24984 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8dc12d34f6346e6b52ffe55952c1e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8dc12d34f6346e6b52ffe55952c1e65" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43307 } } }
I20260620 15:01:43.230290 24984 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:43.230453 24992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:43.233493 24992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:43.238880 24992 catalog_manager.cc:1347] Generated new cluster ID: ab1b81689a1348ec81e99e6df727ec5c
I20260620 15:01:43.238950 24992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:43.245994 24992 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:43.246776 24992 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:43.255201 24992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65: Generated new TSK 0
I20260620 15:01:43.255833 24992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:43.259629 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43307
--builtin_ntp_servers=127.7.46.148:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:01:43.371577 25002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:43.371843 25002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:43.371953 25002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:43.375926 25002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:43.375993 25002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:43.376078 25002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:01:43.380710 25002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:43.381629 25002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:43.382890 25002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:01:43.392001 25002 server_base.cc:1034] running on GCE node
W20260620 15:01:43.391934 25008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:43.391930 25007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:43.391907 25010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:43.392496 25002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:43.393146 25002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:43.394322 25002 hybrid_clock.cc:648] HybridClock initialized: now 1781967703394304 us; error 37 us; skew 500 ppm
I20260620 15:01:43.396159 25002 webserver.cc:460] Webserver started at http://127.7.46.129:43485/ using document root <none> and password file <none>
I20260620 15:01:43.396696 25002 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:43.396782 25002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:43.397006 25002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:43.398881 25002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "20721051fff1b5010129961245dbb314"
server_key_iv: "487dd66ecb5553a24792e0056e5c7b55"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.399420 25002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "20721051fff1b5010129961245dbb314"
server_key_iv: "487dd66ecb5553a24792e0056e5c7b55"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.402750 25002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 15:01:43.405007 25016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.406173 25002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:01:43.406307 25002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "20721051fff1b5010129961245dbb314"
server_key_iv: "487dd66ecb5553a24792e0056e5c7b55"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.406423 25002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:43.436280 25002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:43.436968 25002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:43.437165 25002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:43.437727 25002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:43.438635 25002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:43.438711 25002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.438782 25002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:43.438828 25002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.447405 25002 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:44919
I20260620 15:01:43.447435 25129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:44919 every 8 connection(s)
I20260620 15:01:43.448251 25002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:43.456117 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 25002
I20260620 15:01:43.456238 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:43.456578 7354 external_mini_cluster.cc:1428] Setting key 0a583a7bd5db9f2b2b03bc386ff1993e
I20260620 15:01:43.457517 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.449893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:36701 (local address 127.7.46.190:43307)
0620 15:01:43.450033 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.450038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.451467 (+ 1429us) server_negotiation.cc:407] Connection header received
0620 15:01:43.453008 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.453011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.453054 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.453125 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:43.453896 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.454400 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.455476 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.455692 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.455824 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.456987 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:43.456991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:43.456994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:43.457016 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:43.457037 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:43.457047 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:43.457127 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.457330 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.457385 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:01:43.458443 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.450422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43307 (local address 127.7.46.129:36701)
0620 15:01:43.451326 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.451374 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:43.452826 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:43.453244 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:43.453250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:43.453328 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:43.453785 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.453796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.454508 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.454511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.455378 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.455387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.455891 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.455897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.456102 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.456695 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:43.456698 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:43.456749 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:43.456823 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:43.457158 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:43.457169 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:43.457175 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:01:43.457301 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 15:01:43.457530 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":142,"threads_started":1}
I20260620 15:01:43.458562 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43307
--builtin_ntp_servers=127.7.46.148:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:01:43.459699 25130 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43307
I20260620 15:01:43.459935 25130 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:43.460512 25130 heartbeater.cc:511] Master 127.7.46.190:43307 requested a full tablet report, sending...
I20260620 15:01:43.462077 24942 ts_manager.cc:194] Registered new tserver with Master: 7ed0f3eadeac4fb98eb8d0e598153c0c (127.7.46.129:44919)
I20260620 15:01:43.463836 24942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.129:36701
W20260620 15:01:43.564515 25133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:43.564746 25133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:43.564782 25133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:43.568820 25133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:43.568867 25133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:43.568918 25133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:01:43.573400 25133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:43.574278 25133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:43.575451 25133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:43.584486 25138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:43.584517 25139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:43.584580 25133 server_base.cc:1034] running on GCE node
W20260620 15:01:43.584486 25141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:43.585137 25133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:43.585701 25133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:43.586863 25133 hybrid_clock.cc:648] HybridClock initialized: now 1781967703586844 us; error 29 us; skew 500 ppm
I20260620 15:01:43.588891 25133 webserver.cc:460] Webserver started at http://127.7.46.130:34043/ using document root <none> and password file <none>
I20260620 15:01:43.589370 25133 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:43.589433 25133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:43.589579 25133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:43.591377 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "c90094786f3245a2be669e57d5ac515c"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "78d72465d75b0a83de56352f51d4ce4c"
server_key_iv: "3a0c1cd7f0dc8bb4a18904d4ce96cbe2"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.591935 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "c90094786f3245a2be669e57d5ac515c"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "78d72465d75b0a83de56352f51d4ce4c"
server_key_iv: "3a0c1cd7f0dc8bb4a18904d4ce96cbe2"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.595501 25133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:01:43.597851 25147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.598995 25133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:43.599089 25133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "c90094786f3245a2be669e57d5ac515c"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "78d72465d75b0a83de56352f51d4ce4c"
server_key_iv: "3a0c1cd7f0dc8bb4a18904d4ce96cbe2"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.599169 25133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:43.608963 25133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:43.609485 25133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:43.609622 25133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:43.610147 25133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:43.610966 25133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:43.611012 25133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.611049 25133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:43.611065 25133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.619246 25133 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:35271
I20260620 15:01:43.619310 25260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:35271 every 8 connection(s)
I20260620 15:01:43.620136 25133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:43.624574 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 25133
I20260620 15:01:43.624737 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:43.625046 7354 external_mini_cluster.cc:1428] Setting key 52fd0e4ffd7120a9f47c1f057bfee466
I20260620 15:01:43.626945 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43307
--builtin_ntp_servers=127.7.46.148:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:01:43.629400 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.621811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:43965 (local address 127.7.46.190:43307)
0620 15:01:43.621939 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.621943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.622924 (+ 981us) server_negotiation.cc:407] Connection header received
0620 15:01:43.624323 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.624330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.624399 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.624499 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:43.625468 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.625991 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.627320 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.627734 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.627887 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.628890 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:43.628894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:43.628896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:43.628908 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:43.628935 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:43.628942 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:43.629018 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.629185 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.629263 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:01:43.630259 25263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.622186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43307 (local address 127.7.46.130:43965)
0620 15:01:43.622783 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.622820 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:43.624105 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:43.624645 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:43.624652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:43.624781 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:43.625314 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.625328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.626115 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.626118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.627202 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.627212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.627903 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.627906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.628036 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.628651 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:43.628652 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:43.628688 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:43.628744 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:43.629031 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:43.629051 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:43.629061 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:01:43.629185 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 15:01:43.629441 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260620 15:01:43.631413 25261 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43307
I20260620 15:01:43.631659 25261 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:43.632349 25261 heartbeater.cc:511] Master 127.7.46.190:43307 requested a full tablet report, sending...
I20260620 15:01:43.633433 24942 ts_manager.cc:194] Registered new tserver with Master: c90094786f3245a2be669e57d5ac515c (127.7.46.130:35271)
I20260620 15:01:43.633991 24942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.130:43965
W20260620 15:01:43.734858 25264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:43.735075 25264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:43.735141 25264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:43.739176 25264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:43.739223 25264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:43.739274 25264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:01:43.743963 25264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:43.744866 25264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:43.746210 25264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:43.755262 25269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:43.755239 25270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:43.755477 25264 server_base.cc:1034] running on GCE node
W20260620 15:01:43.755638 25272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:43.756021 25264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:43.756557 25264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:43.757728 25264 hybrid_clock.cc:648] HybridClock initialized: now 1781967703757710 us; error 27 us; skew 500 ppm
I20260620 15:01:43.759459 25264 webserver.cc:460] Webserver started at http://127.7.46.131:33557/ using document root <none> and password file <none>
I20260620 15:01:43.760047 25264 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:43.760108 25264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:43.760265 25264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:43.762147 25264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "e80d39a83c344bab948146ad7a855b04"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "d62b1233cfe915dfdfaf403583b4076b"
server_key_iv: "383ecb0d7de71ad7902c1e1c3173b4c3"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.762682 25264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "e80d39a83c344bab948146ad7a855b04"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "d62b1233cfe915dfdfaf403583b4076b"
server_key_iv: "383ecb0d7de71ad7902c1e1c3173b4c3"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.766075 25264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:01:43.768414 25278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.769529 25264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:43.769621 25264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "e80d39a83c344bab948146ad7a855b04"
format_stamp: "Formatted at 2026-06-20 15:01:43 on dist-test-slave-30t4"
server_key: "d62b1233cfe915dfdfaf403583b4076b"
server_key_iv: "383ecb0d7de71ad7902c1e1c3173b4c3"
server_key_version: "encryptionkey@0"
I20260620 15:01:43.769697 25264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:43.789141 25264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:43.789678 25264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:43.789809 25264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:43.790261 25264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:43.791045 25264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:43.791090 25264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.791126 25264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:43.791145 25264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:43.799852 25264 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:33485
I20260620 15:01:43.799881 25391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:33485 every 8 connection(s)
I20260620 15:01:43.800653 25264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:43.803125 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 25264
I20260620 15:01:43.803220 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:43.803495 7354 external_mini_cluster.cc:1428] Setting key fc013819e5c33ff5f5856a1fa99e2d41
I20260620 15:01:43.809399 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.802229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:53575 (local address 127.7.46.190:43307)
0620 15:01:43.802359 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.802362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.803432 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 15:01:43.805220 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.805222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.805264 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.805354 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:43.806208 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.806714 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.807748 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.808040 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.808119 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.809023 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:43.809026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:43.809028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:43.809038 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:43.809058 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:43.809066 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:43.809123 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.809254 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.809297 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:01:43.810184 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.802603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43307 (local address 127.7.46.131:53575)
0620 15:01:43.803294 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.803337 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:43.805006 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:43.805531 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:43.805542 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:43.805642 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:43.806089 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.806100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.806811 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.806813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.807649 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.807657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.808159 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.808162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.808296 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.808804 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:43.808806 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:43.808841 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:43.808896 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:43.809137 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:43.809143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:43.809147 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 15:01:43.809236 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 15:01:43.809476 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":149,"threads_started":1}
I20260620 15:01:43.811241 25392 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43307
I20260620 15:01:43.811460 25392 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:43.811986 25392 heartbeater.cc:511] Master 127.7.46.190:43307 requested a full tablet report, sending...
I20260620 15:01:43.812906 24942 ts_manager.cc:194] Registered new tserver with Master: e80d39a83c344bab948146ad7a855b04 (127.7.46.131:33485)
I20260620 15:01:43.813475 24942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.131:53575
I20260620 15:01:43.816991 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:01:43.822022 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.818785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.7.46.190:43307)
0620 15:01:43.818898 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.818901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.819073 (+ 172us) server_negotiation.cc:407] Connection header received
0620 15:01:43.819160 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.819162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.819197 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.819258 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:43.819455 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.819998 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.820738 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.820941 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.821011 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.821671 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:43.821674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:43.821676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:43.821685 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:43.821701 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:43.821708 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:43.821763 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.821849 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.821893 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 15:01:43.824572 24942 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:01:43.827409 24942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:01:43.840963 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.836095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:44919 (local address 127.0.0.1:33504)
0620 15:01:43.836613 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.836628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:43.836718 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:43.837156 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:43.837162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:43.837184 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:43.837440 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.837451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.838451 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.838454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.839617 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.839626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.840457 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.840461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.840561 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.840597 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:01:43.840701 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:01:43.840815 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":99,"thread_start_us":89,"threads_started":1}
I20260620 15:01:43.840976 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.836244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.7.46.129:44919)
0620 15:01:43.836721 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.836734 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.836771 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:01:43.836852 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.836856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.837004 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.837155 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:43.837558 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.838319 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.839743 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.840323 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.840450 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.840637 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.840726 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.840816 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":127,"threads_started":1}
I20260620 15:01:43.841316 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.835371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:33485 (local address 127.0.0.1:42564)
0620 15:01:43.835706 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.835749 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:43.835885 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:43.837203 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:43.837206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:43.837222 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:43.837493 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.837503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.838928 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.838931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.839923 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.839935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.840962 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.840966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.841062 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.841083 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:01:43.841127 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 15:01:43.841188 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":107,"thread_start_us":94,"threads_started":1}
I20260620 15:01:43.842530 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.835503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.7.46.131:33485)
0620 15:01:43.836745 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.836761 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.836782 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:01:43.836856 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.836861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.837046 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.837363 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:43.837621 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.838802 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.840061 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.840811 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.841502 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.841670 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.841722 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.842348 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":128,"threads_started":1}
I20260620 15:01:43.843632 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.835662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:35271 (local address 127.0.0.1:33620)
0620 15:01:43.836370 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.836383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:43.836482 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:43.837329 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:43.837332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:43.837345 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:43.837510 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.837520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.838928 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.838931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.840953 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.840961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.843342 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.843345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.843417 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.843443 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 15:01:43.843495 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 15:01:43.843541 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":49,"thread_start_us":76,"threads_started":1}
I20260620 15:01:43.844089 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.836166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.7.46.130:35271)
0620 15:01:43.836745 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.836759 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.836782 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:01:43.836856 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.836861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.837046 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.837195 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:43.837622 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.838787 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.841080 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.841907 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.842020 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.842213 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.843842 (+ 1629us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.843945 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":128,"threads_started":1}
I20260620 15:01:43.844556 25064 tablet_service.cc:1467] Processing CreateTablet for tablet f321545879ec4c948262382557a6e866 (DEFAULT_TABLE table=test-table [id=74df4ea414b34b02b2ac2535a8471c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:43.845235 25326 tablet_service.cc:1467] Processing CreateTablet for tablet f321545879ec4c948262382557a6e866 (DEFAULT_TABLE table=test-table [id=74df4ea414b34b02b2ac2535a8471c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:43.845657 25064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f321545879ec4c948262382557a6e866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:43.846186 25326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f321545879ec4c948262382557a6e866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:43.846594 25195 tablet_service.cc:1467] Processing CreateTablet for tablet f321545879ec4c948262382557a6e866 (DEFAULT_TABLE table=test-table [id=74df4ea414b34b02b2ac2535a8471c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:43.847357 25195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f321545879ec4c948262382557a6e866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:43.852799 25409 tablet_bootstrap.cc:492] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Bootstrap starting.
I20260620 15:01:43.853598 25411 tablet_bootstrap.cc:492] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c: Bootstrap starting.
I20260620 15:01:43.854061 25410 tablet_bootstrap.cc:492] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: Bootstrap starting.
I20260620 15:01:43.855034 25409 tablet_bootstrap.cc:654] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:43.855722 25409 log.cc:824] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:43.855788 25411 tablet_bootstrap.cc:654] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:43.856016 25410 tablet_bootstrap.cc:654] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:43.856627 25411 log.cc:824] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:43.856683 25410 log.cc:824] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:43.858075 25409 tablet_bootstrap.cc:492] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: No bootstrap required, opened a new log
I20260620 15:01:43.858273 25409 ts_tablet_manager.cc:1397] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 15:01:43.858464 25411 tablet_bootstrap.cc:492] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c: No bootstrap required, opened a new log
I20260620 15:01:43.858455 25410 tablet_bootstrap.cc:492] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: No bootstrap required, opened a new log
I20260620 15:01:43.858633 25410 ts_tablet_manager.cc:1397] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:01:43.858633 25411 ts_tablet_manager.cc:1397] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 15:01:43.861429 25409 raft_consensus.cc:348] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.861660 25409 raft_consensus.cc:374] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:43.861732 25409 raft_consensus.cc:729] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c90094786f3245a2be669e57d5ac515c, State: Initialized, Role: FOLLOWER
I20260620 15:01:43.862198 25409 consensus_queue.cc:260] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.862874 25261 heartbeater.cc:503] Master 127.7.46.190:43307 was elected leader, sending a full tablet report...
I20260620 15:01:43.862766 25410 raft_consensus.cc:348] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.862766 25411 raft_consensus.cc:348] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.862998 25411 raft_consensus.cc:374] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:43.862998 25410 raft_consensus.cc:374] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:43.863047 25411 raft_consensus.cc:729] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed0f3eadeac4fb98eb8d0e598153c0c, State: Initialized, Role: FOLLOWER
I20260620 15:01:43.863055 25410 raft_consensus.cc:729] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80d39a83c344bab948146ad7a855b04, State: Initialized, Role: FOLLOWER
I20260620 15:01:43.863448 25410 consensus_queue.cc:260] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.863494 25411 consensus_queue.cc:260] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.864389 25409 ts_tablet_manager.cc:1428] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 15:01:43.864395 25410 ts_tablet_manager.cc:1428] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 15:01:43.864409 25411 ts_tablet_manager.cc:1428] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 15:01:43.864711 25130 heartbeater.cc:503] Master 127.7.46.190:43307 was elected leader, sending a full tablet report...
I20260620 15:01:43.865567 25392 heartbeater.cc:503] Master 127.7.46.190:43307 was elected leader, sending a full tablet report...
W20260620 15:01:43.870981 25262 tablet.cc:2369] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:43.942008 25417 raft_consensus.cc:482] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:43.942261 25417 raft_consensus.cc:504] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.943359 25417 leader_election.cc:288] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e80d39a83c344bab948146ad7a855b04 (127.7.46.131:33485), c90094786f3245a2be669e57d5ac515c (127.7.46.130:35271)
I20260620 15:01:43.948271 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.943680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:50543 (local address 127.7.46.131:33485)
0620 15:01:43.943843 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.943847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.943862 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:01:43.944013 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.944016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.944060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.944156 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:43.944465 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.945257 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.946961 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.947557 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.947671 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.947718 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.948076 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.948145 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:01:43.948271 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.943807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:39243 (local address 127.7.46.130:35271)
0620 15:01:43.943941 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.943946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.944128 (+ 182us) server_negotiation.cc:407] Connection header received
0620 15:01:43.944257 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.944262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.944314 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.944373 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:43.944643 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.945338 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.946960 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.947551 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.947671 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.947718 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.948076 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.948143 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:01:43.948277 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.943586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:33485 (local address 127.7.46.129:50543)
0620 15:01:43.943726 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.943743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:43.943866 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:43.944187 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:43.944191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:43.944210 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:43.944326 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.944332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.945369 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.945372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.946830 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.946841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.947677 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.947681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.947965 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.947985 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 15:01:43.948052 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 15:01:43.948130 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":168}
I20260620 15:01:43.948285 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.943586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:35271 (local address 127.7.46.129:39243)
0620 15:01:43.944046 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:43.944057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:43.944155 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:43.944420 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:43.944423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:43.944443 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:43.944545 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.944549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.945447 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.945451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.946835 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:43.946841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.947679 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.947682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:43.947916 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.947959 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:01:43.948041 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 15:01:43.948130 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":340,"thread_start_us":95,"threads_started":1}
I20260620 15:01:43.949083 25215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f321545879ec4c948262382557a6e866" candidate_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c90094786f3245a2be669e57d5ac515c" is_pre_election: true
I20260620 15:01:43.949123 25346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f321545879ec4c948262382557a6e866" candidate_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80d39a83c344bab948146ad7a855b04" is_pre_election: true
I20260620 15:01:43.949398 25215 raft_consensus.cc:2454] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed0f3eadeac4fb98eb8d0e598153c0c in term 0.
W20260620 15:01:43.949401 25131 tablet.cc:2369] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:43.949402 25346 raft_consensus.cc:2454] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed0f3eadeac4fb98eb8d0e598153c0c in term 0.
I20260620 15:01:43.949863 25017 leader_election.cc:302] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed0f3eadeac4fb98eb8d0e598153c0c, c90094786f3245a2be669e57d5ac515c; no voters:
I20260620 15:01:43.950171 25417 raft_consensus.cc:2781] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:01:43.950246 25417 raft_consensus.cc:482] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:01:43.950276 25417 raft_consensus.cc:3037] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:43.951290 25417 raft_consensus.cc:504] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.951656 25417 leader_election.cc:288] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [CANDIDATE]: Term 1 election: Requested vote from peers e80d39a83c344bab948146ad7a855b04 (127.7.46.131:33485), c90094786f3245a2be669e57d5ac515c (127.7.46.130:35271)
I20260620 15:01:43.951925 25346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f321545879ec4c948262382557a6e866" candidate_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80d39a83c344bab948146ad7a855b04"
I20260620 15:01:43.952065 25346 raft_consensus.cc:3037] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:43.952083 25215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f321545879ec4c948262382557a6e866" candidate_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c90094786f3245a2be669e57d5ac515c"
I20260620 15:01:43.952184 25215 raft_consensus.cc:3037] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:43.953328 25346 raft_consensus.cc:2454] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed0f3eadeac4fb98eb8d0e598153c0c in term 1.
I20260620 15:01:43.953375 25215 raft_consensus.cc:2454] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed0f3eadeac4fb98eb8d0e598153c0c in term 1.
I20260620 15:01:43.953606 25017 leader_election.cc:302] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed0f3eadeac4fb98eb8d0e598153c0c, c90094786f3245a2be669e57d5ac515c; no voters:
I20260620 15:01:43.953754 25417 raft_consensus.cc:2781] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:43.953971 25417 raft_consensus.cc:686] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [term 1 LEADER]: Becoming Leader. State: Replica: 7ed0f3eadeac4fb98eb8d0e598153c0c, State: Running, Role: LEADER
I20260620 15:01:43.954272 25417 consensus_queue.cc:237] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } }
I20260620 15:01:43.957175 24940 catalog_manager.cc:5495] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ed0f3eadeac4fb98eb8d0e598153c0c (127.7.46.129). New cstate: current_term: 1 leader_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed0f3eadeac4fb98eb8d0e598153c0c" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 44919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 } health_report { overall_health: UNKNOWN } } }
I20260620 15:01:43.990597 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:43.987138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.7.46.129:44919)
0620 15:01:43.987293 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:43.987298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:43.987311 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:01:43.987384 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:43.987386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:43.987439 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:43.987548 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:43.987809 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.988320 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.989171 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:43.989391 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:43.989499 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:43.989792 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:43.990183 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:01:43.990311 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:43.990372 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 15:01:43.990424 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:01:43.997097 25215 raft_consensus.cc:1261] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 1 FOLLOWER]: Refusing update from remote peer 7ed0f3eadeac4fb98eb8d0e598153c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:43.997112 25346 raft_consensus.cc:1261] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 1 FOLLOWER]: Refusing update from remote peer 7ed0f3eadeac4fb98eb8d0e598153c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:43.997767 25417 consensus_queue.cc:1035] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c90094786f3245a2be669e57d5ac515c" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:43.998001 25419 consensus_queue.cc:1035] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e80d39a83c344bab948146ad7a855b04" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:44.004570 25422 mvcc.cc:205] Tried to move back new op lower bound from 7298939715564314624 to 7298939715399311360. Current Snapshot: MvccSnapshot[applied={T|T < 7298939715564314624}]
I20260620 15:01:44.006506 25423 mvcc.cc:205] Tried to move back new op lower bound from 7298939715564314624 to 7298939715399311360. Current Snapshot: MvccSnapshot[applied={T|T < 7298939715564314624}]
I20260620 15:01:44.008610 25424 mvcc.cc:205] Tried to move back new op lower bound from 7298939715564314624 to 7298939715399311360. Current Snapshot: MvccSnapshot[applied={T|T < 7298939715564314624}]
I20260620 15:01:44.012777 24940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:44.012903 24940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:44.015318 24940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8dc12d34f6346e6b52ffe55952c1e65: Sending DeleteTablet for 3 replicas of tablet f321545879ec4c948262382557a6e866
I20260620 15:01:44.016134 25195 tablet_service.cc:1514] Processing DeleteTablet for tablet f321545879ec4c948262382557a6e866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20260620 15:01:44.016165 25326 tablet_service.cc:1514] Processing DeleteTablet for tablet f321545879ec4c948262382557a6e866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:44 UTC) from {username='slave'} at 127.0.0.1:42564
I20260620 15:01:44.016634 25433 tablet_replica.cc:330] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: stopping tablet replica
I20260620 15:01:44.016764 25064 tablet_service.cc:1514] Processing DeleteTablet for tablet f321545879ec4c948262382557a6e866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:44 UTC) from {username='slave'} at 127.0.0.1:33504
I20260620 15:01:44.016995 25433 raft_consensus.cc:2229] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:44.017268 25434 tablet_replica.cc:330] T f321545879ec4c948262382557a6e866 P 7ed0f3eadeac4fb98eb8d0e598153c0c: stopping tablet replica
I20260620 15:01:44.017330 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 25002
I20260620 15:01:44.017352 25433 raft_consensus.cc:2258] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:44.018158 25432 tablet_replica.cc:330] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: stopping tablet replica
I20260620 15:01:44.018424 25432 raft_consensus.cc:2229] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:44.018699 25432 raft_consensus.cc:2258] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:44.019064 25433 ts_tablet_manager.cc:1905] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:44.020831 25432 ts_tablet_manager.cc:1905] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:44.022471 25433 ts_tablet_manager.cc:1918] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:44.022569 25433 log.cc:1196] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/f321545879ec4c948262382557a6e866
I20260620 15:01:44.022863 25432 ts_tablet_manager.cc:1918] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:44.022930 25433 ts_tablet_manager.cc:1939] T f321545879ec4c948262382557a6e866 P e80d39a83c344bab948146ad7a855b04: Deleting consensus metadata
I20260620 15:01:44.022972 25432 log.cc:1196] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/f321545879ec4c948262382557a6e866
I20260620 15:01:44.023244 25432 ts_tablet_manager.cc:1939] T f321545879ec4c948262382557a6e866 P c90094786f3245a2be669e57d5ac515c: Deleting consensus metadata
I20260620 15:01:44.024173 24928 catalog_manager.cc:4841] TS e80d39a83c344bab948146ad7a855b04 (127.7.46.131:33485): tablet f321545879ec4c948262382557a6e866 (table test-table [id=74df4ea414b34b02b2ac2535a8471c90]) successfully deleted
I20260620 15:01:44.024241 24926 catalog_manager.cc:4841] TS c90094786f3245a2be669e57d5ac515c (127.7.46.130:35271): tablet f321545879ec4c948262382557a6e866 (table test-table [id=74df4ea414b34b02b2ac2535a8471c90]) successfully deleted
W20260620 15:01:44.025238 24928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:44919 (error 108)
I20260620 15:01:44.025512 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 25133
I20260620 15:01:44.027525 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:44.027233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:44919 (local address 127.0.0.1:33514)
0620 15:01:44.027356 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:44.027445 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:44919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:01:44.027705 24928 catalog_manager.cc:4584] TS 7ed0f3eadeac4fb98eb8d0e598153c0c (127.7.46.129:44919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f321545879ec4c948262382557a6e866: Network error: Client connection negotiation failed: client connection to 127.7.46.129:44919: connect: Connection refused (error 111)
I20260620 15:01:44.031680 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 25264
I20260620 15:01:44.037146 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 24911
2026-06-20T15:01:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1014 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:44 dist-test-slave-30t4 krb5kdc[25438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:44 dist-test-slave-30t4 krb5kdc[25438](info): set up 2 sockets
Jun 20 15:01:44 dist-test-slave-30t4 krb5kdc[25438](info): commencing operation
krb5kdc: starting...
W20260620 15:01:45.087618 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:45 dist-test-slave-30t4 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:01:45.205997 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:41369
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:41369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:45.311918 25451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:45.312166 25451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:45.312240 25451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:45.316102 25451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:45.316179 25451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:45.316216 25451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:45.316272 25451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:45.316308 25451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:45.321329 25451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:41369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:45.322255 25451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:45.323184 25451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:45.331036 25457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:45.331056 25459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:45.331194 25451 server_base.cc:1034] running on GCE node
W20260620 15:01:45.331056 25456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:45.331878 25451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:45.332881 25451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:45.334071 25451 hybrid_clock.cc:648] HybridClock initialized: now 1781967705334038 us; error 43 us; skew 500 ppm
Jun 20 15:01:45 dist-test-slave-30t4 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967705, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:45.339989 25451 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:01:45.340893 25451 webserver.cc:460] Webserver started at http://127.7.46.190:40061/ using document root <none> and password file <none>
I20260620 15:01:45.341473 25451 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:45.341559 25451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:45.341770 25451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:45.343698 25451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "54a3e42d3af6407aa28cba19264d66b5"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "17a0a81a68195be7eb44a64d11dd5386"
server_key_iv: "c7ee62e8481d0c85fd56bda2d1c752c6"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.344305 25451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "54a3e42d3af6407aa28cba19264d66b5"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "17a0a81a68195be7eb44a64d11dd5386"
server_key_iv: "c7ee62e8481d0c85fd56bda2d1c752c6"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.347745 25451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:45.350039 25466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:45.351212 25451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:45.351342 25451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "54a3e42d3af6407aa28cba19264d66b5"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "17a0a81a68195be7eb44a64d11dd5386"
server_key_iv: "c7ee62e8481d0c85fd56bda2d1c752c6"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.351445 25451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:45.367515 25451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:45.370038 25451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:45.370252 25451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:45.377095 25451 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:41369
I20260620 15:01:45.377099 25518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:41369 every 8 connection(s)
I20260620 15:01:45.377945 25451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:45.380491 25519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:45.381181 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 25451
I20260620 15:01:45.381319 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:45.381628 7354 external_mini_cluster.cc:1428] Setting key 3d8a8230423371cdc16e8c673bf779ac
I20260620 15:01:45.386281 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5: Bootstrap starting.
I20260620 15:01:45.388347 25519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:45.389029 25519 log.cc:824] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:01:45 dist-test-slave-30t4 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:45.390959 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5: No bootstrap required, opened a new log
I20260620 15:01:45.395457 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:45.382876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.7.46.190:41369)
0620 15:01:45.383315 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:45.383338 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:45.383381 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:01:45.384207 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:45.384226 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:45.384535 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:45.384844 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:45.385589 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.386570 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.387428 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.387933 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.388097 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:45.390567 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:45.390573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:45.390586 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:45.390614 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:45.392572 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:45.392980 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:45.392988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:45.392995 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:45.393151 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:45.393402 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:45.393406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:45.393408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:45.393878 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:45.394057 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:45.394289 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 15:01:45.394565 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
I20260620 15:01:45.395541 25519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a3e42d3af6407aa28cba19264d66b5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41369 } }
I20260620 15:01:45.395795 25519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:45.395854 25519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a3e42d3af6407aa28cba19264d66b5, State: Initialized, Role: FOLLOWER
I20260620 15:01:45.396322 25519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a3e42d3af6407aa28cba19264d66b5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41369 } }
I20260620 15:01:45.396483 25519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:45.396552 25519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:45.396646 25519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:45.397471 25519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a3e42d3af6407aa28cba19264d66b5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41369 } }
I20260620 15:01:45.397773 25519 leader_election.cc:302] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54a3e42d3af6407aa28cba19264d66b5; no voters:
I20260620 15:01:45.398042 25519 leader_election.cc:288] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:45.398155 25524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:45.398376 25524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [term 1 LEADER]: Becoming Leader. State: Replica: 54a3e42d3af6407aa28cba19264d66b5, State: Running, Role: LEADER
I20260620 15:01:45.398729 25524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a3e42d3af6407aa28cba19264d66b5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41369 } }
I20260620 15:01:45.399134 25519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:45.400384 25526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54a3e42d3af6407aa28cba19264d66b5. Latest consensus state: current_term: 1 leader_uuid: "54a3e42d3af6407aa28cba19264d66b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a3e42d3af6407aa28cba19264d66b5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41369 } } }
I20260620 15:01:45.400436 25525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54a3e42d3af6407aa28cba19264d66b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a3e42d3af6407aa28cba19264d66b5" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41369 } } }
I20260620 15:01:45.400511 25526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:45.400537 25525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:45.400839 25533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:45.403990 25533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:45.409407 25533 catalog_manager.cc:1347] Generated new cluster ID: 92041ec4daae464083ea42276b3fe353
I20260620 15:01:45.409494 25533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:45.426406 25533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:45.427296 25533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:45.435146 25533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5: Generated new TSK 0
I20260620 15:01:45.435810 25533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 15:01:45.471424 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41369
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:45.592690 25545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:45.592911 25545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:45.592949 25545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:45.596865 25545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:45.596947 25545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:45.597039 25545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:01:45.602123 25545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:45.602977 25545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:45.603842 25545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:45.613067 25551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:45.613086 25553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:45.613137 25545 server_base.cc:1034] running on GCE node
W20260620 15:01:45.613072 25550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:45.613700 25545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:45.614328 25545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:45.615518 25545 hybrid_clock.cc:648] HybridClock initialized: now 1781967705615496 us; error 32 us; skew 500 ppm
Jun 20 15:01:45 dist-test-slave-30t4 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967705, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:45.618307 25545 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:01:45.619047 25545 webserver.cc:460] Webserver started at http://127.7.46.129:44031/ using document root <none> and password file <none>
I20260620 15:01:45.619524 25545 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:45.619581 25545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:45.619730 25545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:45.621681 25545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "31c261dc16ad486c93869483aafc82f1"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "e9bbd644ba7795460f011dd1d5311f4d"
server_key_iv: "cf2af961b80e6303114d38ec341d332c"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.622241 25545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "31c261dc16ad486c93869483aafc82f1"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "e9bbd644ba7795460f011dd1d5311f4d"
server_key_iv: "cf2af961b80e6303114d38ec341d332c"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.625756 25545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:01:45.628082 25560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:45.629179 25545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:01:45.629304 25545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "31c261dc16ad486c93869483aafc82f1"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "e9bbd644ba7795460f011dd1d5311f4d"
server_key_iv: "cf2af961b80e6303114d38ec341d332c"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.629401 25545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:45.671700 25545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:45.674924 25545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:45.675122 25545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:45.675697 25545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:45.676559 25545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:45.676605 25545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:45.676644 25545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:45.676659 25545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:45.685137 25545 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:34645
I20260620 15:01:45.685199 25673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:34645 every 8 connection(s)
I20260620 15:01:45.686002 25545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:45.688301 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 25545
I20260620 15:01:45.688462 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:45.688788 7354 external_mini_cluster.cc:1428] Setting key c391fc6e905dbf6c252b37fbff1b3567
Jun 20 15:01:45 dist-test-slave-30t4 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967705, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 15:01:45.699906 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:45.687759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:47229 (local address 127.7.46.190:41369)
0620 15:01:45.687918 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:45.687922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:45.688757 (+ 835us) server_negotiation.cc:407] Connection header received
0620 15:01:45.690293 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:45.690296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:45.690343 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:45.690428 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:45.691486 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.692193 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.693604 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.693848 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.693961 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:45.696275 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:45.696279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:45.696282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:45.696312 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:45.697759 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:45.698251 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:45.698255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:45.698257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:45.698336 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:45.698701 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:45.698709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:45.698710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:45.698941 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:45.699048 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:45.699602 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 15:01:45.699720 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 15:01:45.700717 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:45.688161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:41369 (local address 127.7.46.129:47229)
0620 15:01:45.688595 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:45.688645 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:45.690045 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:45.690568 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:45.690574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:45.690894 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:45.691340 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:45.691352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.692316 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.692321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:45.693478 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:45.693488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.693996 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.693999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:45.694174 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:45.694724 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:45.694739 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:45.696118 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:45.697877 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:45.697883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:45.697895 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:45.698146 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:45.698430 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:45.698434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:45.698436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:45.698570 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:45.699085 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:45.699091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:45.699309 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 15:01:45.699492 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 15:01:45.699822 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260620 15:01:45.701850 25674 heartbeater.cc:347] Connected to a master server at 127.7.46.190:41369
I20260620 15:01:45.702071 25674 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:45.702518 25674 heartbeater.cc:511] Master 127.7.46.190:41369 requested a full tablet report, sending...
I20260620 15:01:45.703982 25483 ts_manager.cc:194] Registered new tserver with Master: 31c261dc16ad486c93869483aafc82f1 (127.7.46.129:34645)
I20260620 15:01:45.705061 25483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:47229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 15:01:45.714764 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41369
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:45.820117 25679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:45.820327 25679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:45.820369 25679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:45.824213 25679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:45.824260 25679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:45.824313 25679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:01:45.829295 25679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:45.830119 25679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:45.830993 25679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:45.840262 25685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:45.840297 25684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:45.840354 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:45.840392 25679 server_base.cc:1034] running on GCE node
I20260620 15:01:45.840881 25679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:45.841499 25679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:45.842684 25679 hybrid_clock.cc:648] HybridClock initialized: now 1781967705842646 us; error 58 us; skew 500 ppm
Jun 20 15:01:45 dist-test-slave-30t4 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967705, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:45.845402 25679 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:01:45.846216 25679 webserver.cc:460] Webserver started at http://127.7.46.130:45607/ using document root <none> and password file <none>
I20260620 15:01:45.846751 25679 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:45.846843 25679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:45.847062 25679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:45.849056 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "ce6d1f7d25844f97b257ce5703cf6a3d"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "8eccea6351111a7dec29e8707aa2a7ad"
server_key_iv: "9c0cb2c3745e740ca4be89666b3ad913"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.849604 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "ce6d1f7d25844f97b257ce5703cf6a3d"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "8eccea6351111a7dec29e8707aa2a7ad"
server_key_iv: "9c0cb2c3745e740ca4be89666b3ad913"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.853081 25679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:45.855437 25694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:45.856671 25679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:01:45.856804 25679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "ce6d1f7d25844f97b257ce5703cf6a3d"
format_stamp: "Formatted at 2026-06-20 15:01:45 on dist-test-slave-30t4"
server_key: "8eccea6351111a7dec29e8707aa2a7ad"
server_key_iv: "9c0cb2c3745e740ca4be89666b3ad913"
server_key_version: "encryptionkey@0"
I20260620 15:01:45.856909 25679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:45.874410 25679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:45.877514 25679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:45.877707 25679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:45.878317 25679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:45.879102 25679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:45.879173 25679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:45.879244 25679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:45.879295 25679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:45.888077 25679 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:44153
I20260620 15:01:45.888095 25807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:44153 every 8 connection(s)
I20260620 15:01:45.888896 25679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:45.890200 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 25679
I20260620 15:01:45.890336 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:45.890686 7354 external_mini_cluster.cc:1428] Setting key a4e6c0497b3b3057c603c25a50888d87
Jun 20 15:01:45 dist-test-slave-30t4 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967705, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
I20260620 15:01:45.902459 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:45.892612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:56707 (local address 127.7.46.190:41369)
0620 15:01:45.892751 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:45.892756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:45.892772 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:01:45.892879 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:45.892882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:45.892926 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:45.893007 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:45.894112 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.894629 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.895759 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.896035 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.896151 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:45.898700 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:45.898706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:45.898714 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:45.898736 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:45.900348 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:45.900902 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:45.900906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:45.900909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:45.900995 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:45.901322 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:45.901325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:45.901326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:45.901554 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:45.901679 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:45.902126 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 15:01:45.902268 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:01:45.903200 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:45.891094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:41369 (local address 127.7.46.130:56707)
0620 15:01:45.891598 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:45.891664 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:45.892688 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:45.893184 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:45.893191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:45.893518 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:45.893964 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:45.893977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.894753 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.894756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:45.895628 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:45.895638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:45.896184 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:45.896188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:45.896368 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:45.896929 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:45.896945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:45.898545 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:45.900468 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:45.900475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:45.900488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:45.900774 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:45.901087 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:45.901090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:45.901091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:45.901212 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:45.901658 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:45.901663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:45.901913 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 15:01:45.902065 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:01:45.902334 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":128,"threads_started":1}
I20260620 15:01:45.904758 25808 heartbeater.cc:347] Connected to a master server at 127.7.46.190:41369
I20260620 15:01:45.904999 25808 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:45.905463 25808 heartbeater.cc:511] Master 127.7.46.190:41369 requested a full tablet report, sending...
I20260620 15:01:45.906474 25483 ts_manager.cc:194] Registered new tserver with Master: ce6d1f7d25844f97b257ce5703cf6a3d (127.7.46.130:44153)
I20260620 15:01:45.907032 25483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:56707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 15:01:45.919147 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41369
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:46.027675 25813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:46.027925 25813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:46.027992 25813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:46.031888 25813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:46.031934 25813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:46.032037 25813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:01:46.037283 25813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:46.038184 25813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:46.039114 25813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:46.048223 25818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:46.048281 25813 server_base.cc:1034] running on GCE node
W20260620 15:01:46.048230 25819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:46.048230 25821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:46.048753 25813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:46.049364 25813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:46.050559 25813 hybrid_clock.cc:648] HybridClock initialized: now 1781967706050550 us; error 30 us; skew 500 ppm
Jun 20 15:01:46 dist-test-slave-30t4 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967706, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:46.053158 25813 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:01:46.053953 25813 webserver.cc:460] Webserver started at http://127.7.46.131:43901/ using document root <none> and password file <none>
I20260620 15:01:46.054522 25813 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:46.054586 25813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:46.054786 25813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:46.056777 25813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "1392e4d0aa074c029b5c6a66779726eb"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "97adebb09ec5e2314e6c564ef4d5796b"
server_key_iv: "f8747d694c457cebb3d0b7e29c5a0f64"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.057264 25813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "1392e4d0aa074c029b5c6a66779726eb"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "97adebb09ec5e2314e6c564ef4d5796b"
server_key_iv: "f8747d694c457cebb3d0b7e29c5a0f64"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.060755 25813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:01:46.063237 25828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.064469 25813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:01:46.064574 25813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "1392e4d0aa074c029b5c6a66779726eb"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "97adebb09ec5e2314e6c564ef4d5796b"
server_key_iv: "f8747d694c457cebb3d0b7e29c5a0f64"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.064656 25813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:46.082719 25813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:46.085404 25813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:46.085554 25813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:46.086033 25813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:46.086753 25813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:46.086799 25813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.086838 25813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:46.086853 25813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.095839 25813 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:35411
I20260620 15:01:46.095840 25941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:35411 every 8 connection(s)
I20260620 15:01:46.096724 25813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:46.105054 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 25813
I20260620 15:01:46.105190 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:46.105551 7354 external_mini_cluster.cc:1428] Setting key bd87c19ab4efc81b64467c64deff5341
Jun 20 15:01:46 dist-test-slave-30t4 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967706, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:46.109710 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.098416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:39511 (local address 127.7.46.190:41369)
0620 15:01:46.098545 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.098548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.099560 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 15:01:46.100565 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.100568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.100626 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.100732 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:46.101864 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.102394 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.103541 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.103756 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.103917 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.106477 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:46.106481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:46.106484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:46.106517 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:46.108094 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:46.108531 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:46.108534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:46.108535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:46.108601 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:46.108873 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:46.108875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:46.108876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:46.109028 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:46.109098 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.109509 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.109599 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:01:46.110482 25944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.098722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:41369 (local address 127.7.46.131:39511)
0620 15:01:46.099412 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.099451 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:46.100351 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:46.100921 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:46.100936 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:46.101264 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:46.101718 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.101730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.102523 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.102527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.103396 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.103405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.103948 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.103952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.104123 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.104697 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:46.104712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:46.106347 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:46.108190 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:46.108195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:46.108204 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:46.108444 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:46.108676 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:46.108678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:46.108679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:46.108800 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:46.109109 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:46.109113 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:46.109314 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 15:01:46.109458 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 15:01:46.109687 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":103,"threads_started":1}
I20260620 15:01:46.111548 25942 heartbeater.cc:347] Connected to a master server at 127.7.46.190:41369
I20260620 15:01:46.111822 25942 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:46.112262 25942 heartbeater.cc:511] Master 127.7.46.190:41369 requested a full tablet report, sending...
I20260620 15:01:46.113202 25483 ts_manager.cc:194] Registered new tserver with Master: 1392e4d0aa074c029b5c6a66779726eb (127.7.46.131:35411)
I20260620 15:01:46.113740 25483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:39511
I20260620 15:01:46.118930 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:01:46.127588 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.120762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.7.46.190:41369)
0620 15:01:46.120908 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.120911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.121108 (+ 197us) server_negotiation.cc:407] Connection header received
0620 15:01:46.121196 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.121198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.121230 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.121293 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:46.121927 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.122397 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.123237 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.123431 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.123508 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.124907 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:46.124910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:46.124911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:46.124931 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:46.126086 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:46.126533 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:46.126536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:46.126537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:46.126617 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:46.126876 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:46.126878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:46.126879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:46.127051 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:46.127157 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.127365 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.127432 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:01:46.130144 25483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:01:46.133183 25483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:01:46.146885 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.141912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.7.46.129:34645)
0620 15:01:46.142472 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.142487 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.142505 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:01:46.142590 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.142594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.142772 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.142904 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:46.143083 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.143844 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.145378 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.146209 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.146326 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.146486 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.146647 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.146735 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":128,"threads_started":1}
I20260620 15:01:46.146885 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.141521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.7.46.130:44153)
0620 15:01:46.142194 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.142204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.142224 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:01:46.142296 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.142301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.142473 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.142611 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:46.142802 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.143909 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.145583 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.146343 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.146437 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.146553 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.146646 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.146735 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":140,"threads_started":1}
I20260620 15:01:46.146919 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.140835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:44153 (local address 127.0.0.1:34524)
0620 15:01:46.141227 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.141251 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:46.141375 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:46.142597 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:46.142600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:46.142618 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:46.142706 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.142711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.144049 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.144053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.145457 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.145468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.146447 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.146450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.146523 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.146539 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 15:01:46.146592 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 15:01:46.146765 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":138,"spinlock_wait_cycles":5248,"thread_start_us":79,"threads_started":1}
I20260620 15:01:46.146919 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.141480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:34645 (local address 127.0.0.1:46850)
0620 15:01:46.141774 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.141787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:46.141874 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:46.142880 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:46.142882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:46.142898 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:46.142989 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.142994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.143972 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.143978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.145261 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.145270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.146328 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.146330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.146444 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.146487 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:01:46.146604 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 15:01:46.146761 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":89,"thread_start_us":104,"threads_started":1}
I20260620 15:01:46.148572 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.140762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35411 (local address 127.0.0.1:59272)
0620 15:01:46.141193 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.141243 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:46.141375 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:46.142312 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:46.142317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:46.142339 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:46.142473 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.142480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.144049 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.144053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.145462 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.145468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.148172 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.148175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.148263 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.148285 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:01:46.148366 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 15:01:46.148432 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":199,"spinlock_wait_cycles":6144,"thread_start_us":77,"threads_started":1}
I20260620 15:01:46.148825 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.140897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.7.46.131:35411)
0620 15:01:46.141663 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.141676 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.141698 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:01:46.141772 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.141776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.141986 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.142140 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:46.142638 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.143887 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.147058 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.147887 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.147989 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.148110 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.148422 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.148527 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":92,"threads_started":1}
I20260620 15:01:46.149863 25608 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc7685f7bd5478a88f3f94813f0806f (DEFAULT_TABLE table=test-table [id=273f9d6caf1743d5b7d98caf2c0adf6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:46.149884 25742 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc7685f7bd5478a88f3f94813f0806f (DEFAULT_TABLE table=test-table [id=273f9d6caf1743d5b7d98caf2c0adf6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:46.150787 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc7685f7bd5478a88f3f94813f0806f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:46.150851 25742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc7685f7bd5478a88f3f94813f0806f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:46.151163 25876 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc7685f7bd5478a88f3f94813f0806f (DEFAULT_TABLE table=test-table [id=273f9d6caf1743d5b7d98caf2c0adf6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:46.152068 25876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc7685f7bd5478a88f3f94813f0806f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:46.156831 25959 tablet_bootstrap.cc:492] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1: Bootstrap starting.
I20260620 15:01:46.158520 25961 tablet_bootstrap.cc:492] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: Bootstrap starting.
I20260620 15:01:46.158975 25959 tablet_bootstrap.cc:654] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:46.159065 25960 tablet_bootstrap.cc:492] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: Bootstrap starting.
I20260620 15:01:46.160444 25959 log.cc:824] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:46.160586 25961 tablet_bootstrap.cc:654] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:46.161227 25960 tablet_bootstrap.cc:654] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:46.161842 25961 log.cc:824] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:46.162353 25960 log.cc:824] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:46.163682 25961 tablet_bootstrap.cc:492] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: No bootstrap required, opened a new log
I20260620 15:01:46.163892 25961 ts_tablet_manager.cc:1397] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 15:01:46.164211 25959 tablet_bootstrap.cc:492] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1: No bootstrap required, opened a new log
I20260620 15:01:46.164366 25959 ts_tablet_manager.cc:1397] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 15:01:46.167650 25961 raft_consensus.cc:348] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.167971 25961 raft_consensus.cc:374] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:46.168118 25961 raft_consensus.cc:729] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1392e4d0aa074c029b5c6a66779726eb, State: Initialized, Role: FOLLOWER
I20260620 15:01:46.167989 25959 raft_consensus.cc:348] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.168315 25959 raft_consensus.cc:374] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:46.168391 25959 raft_consensus.cc:729] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c261dc16ad486c93869483aafc82f1, State: Initialized, Role: FOLLOWER
I20260620 15:01:46.168556 25961 consensus_queue.cc:260] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.168998 25959 consensus_queue.cc:260] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.169165 25960 tablet_bootstrap.cc:492] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: No bootstrap required, opened a new log
I20260620 15:01:46.169328 25960 ts_tablet_manager.cc:1397] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 15:01:46.169485 25961 ts_tablet_manager.cc:1428] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 15:01:46.169751 25674 heartbeater.cc:503] Master 127.7.46.190:41369 was elected leader, sending a full tablet report...
I20260620 15:01:46.169916 25959 ts_tablet_manager.cc:1428] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 15:01:46.170733 25942 heartbeater.cc:503] Master 127.7.46.190:41369 was elected leader, sending a full tablet report...
I20260620 15:01:46.173156 25960 raft_consensus.cc:348] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.173424 25960 raft_consensus.cc:374] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:46.173482 25960 raft_consensus.cc:729] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6d1f7d25844f97b257ce5703cf6a3d, State: Initialized, Role: FOLLOWER
I20260620 15:01:46.173895 25960 consensus_queue.cc:260] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.174491 25808 heartbeater.cc:503] Master 127.7.46.190:41369 was elected leader, sending a full tablet report...
I20260620 15:01:46.174801 25960 ts_tablet_manager.cc:1428] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 15:01:46.187196 25675 tablet.cc:2369] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:46.210186 25966 raft_consensus.cc:482] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:46.210382 25966 raft_consensus.cc:504] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.211560 25966 leader_election.cc:288] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1392e4d0aa074c029b5c6a66779726eb (127.7.46.131:35411), ce6d1f7d25844f97b257ce5703cf6a3d (127.7.46.130:44153)
I20260620 15:01:46.215253 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.211887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:34705 (local address 127.7.46.131:35411)
0620 15:01:46.212042 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.212047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.212155 (+ 108us) server_negotiation.cc:407] Connection header received
0620 15:01:46.212315 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.212318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.212368 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.212446 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:46.212734 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.213229 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.214280 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.214668 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.214793 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.214845 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.215075 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.215138 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:01:46.215271 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.211723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:35411 (local address 127.7.46.129:34705)
0620 15:01:46.212053 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.212068 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:46.212200 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:46.212467 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:46.212469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:46.212486 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:46.212623 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.212628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.213336 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.213342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.214157 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.214165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.214785 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.214788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.214886 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.214962 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 15:01:46.215049 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 15:01:46.215123 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":45,"thread_start_us":98,"threads_started":1}
I20260620 15:01:46.215313 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.211975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:56359 (local address 127.7.46.130:44153)
0620 15:01:46.212107 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.212111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.212309 (+ 198us) server_negotiation.cc:407] Connection header received
0620 15:01:46.212461 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.212464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.212516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.212606 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:46.212870 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.213344 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.214369 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.214763 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.214889 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.214940 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.215141 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.215201 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:01:46.215337 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.211823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:44153 (local address 127.7.46.129:56359)
0620 15:01:46.212212 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.212223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:46.212340 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:46.212626 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:46.212629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:46.212644 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:46.212763 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.212768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.213459 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.213465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.214244 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.214252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.214880 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.214883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.215012 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.215039 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 15:01:46.215124 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:01:46.215190 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260620 15:01:46.215852 25762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc7685f7bd5478a88f3f94813f0806f" candidate_uuid: "31c261dc16ad486c93869483aafc82f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" is_pre_election: true
I20260620 15:01:46.215855 25896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc7685f7bd5478a88f3f94813f0806f" candidate_uuid: "31c261dc16ad486c93869483aafc82f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1392e4d0aa074c029b5c6a66779726eb" is_pre_election: true
I20260620 15:01:46.216149 25896 raft_consensus.cc:2454] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c261dc16ad486c93869483aafc82f1 in term 0.
I20260620 15:01:46.216149 25762 raft_consensus.cc:2454] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c261dc16ad486c93869483aafc82f1 in term 0.
I20260620 15:01:46.216636 25563 leader_election.cc:302] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1392e4d0aa074c029b5c6a66779726eb, 31c261dc16ad486c93869483aafc82f1, ce6d1f7d25844f97b257ce5703cf6a3d; no voters:
I20260620 15:01:46.216887 25966 raft_consensus.cc:2781] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:01:46.216982 25966 raft_consensus.cc:482] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:01:46.217051 25966 raft_consensus.cc:3037] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:46.218014 25966 raft_consensus.cc:504] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.218344 25966 leader_election.cc:288] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [CANDIDATE]: Term 1 election: Requested vote from peers 1392e4d0aa074c029b5c6a66779726eb (127.7.46.131:35411), ce6d1f7d25844f97b257ce5703cf6a3d (127.7.46.130:44153)
I20260620 15:01:46.218675 25896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc7685f7bd5478a88f3f94813f0806f" candidate_uuid: "31c261dc16ad486c93869483aafc82f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1392e4d0aa074c029b5c6a66779726eb"
I20260620 15:01:46.218693 25762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc7685f7bd5478a88f3f94813f0806f" candidate_uuid: "31c261dc16ad486c93869483aafc82f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d"
I20260620 15:01:46.218803 25762 raft_consensus.cc:3037] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:46.218797 25896 raft_consensus.cc:3037] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:46.219959 25762 raft_consensus.cc:2454] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c261dc16ad486c93869483aafc82f1 in term 1.
I20260620 15:01:46.220088 25896 raft_consensus.cc:2454] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c261dc16ad486c93869483aafc82f1 in term 1.
I20260620 15:01:46.220249 25563 leader_election.cc:302] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c261dc16ad486c93869483aafc82f1, ce6d1f7d25844f97b257ce5703cf6a3d; no voters:
I20260620 15:01:46.220479 25966 raft_consensus.cc:2781] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:46.220657 25966 raft_consensus.cc:686] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [term 1 LEADER]: Becoming Leader. State: Replica: 31c261dc16ad486c93869483aafc82f1, State: Running, Role: LEADER
I20260620 15:01:46.220952 25966 consensus_queue.cc:237] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } }
I20260620 15:01:46.223920 25481 catalog_manager.cc:5495] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31c261dc16ad486c93869483aafc82f1 (127.7.46.129). New cstate: current_term: 1 leader_uuid: "31c261dc16ad486c93869483aafc82f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31c261dc16ad486c93869483aafc82f1" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 34645 } health_report { overall_health: HEALTHY } } }
I20260620 15:01:46.233887 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.230348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.7.46.129:34645)
0620 15:01:46.230472 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.230478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.230491 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:01:46.230656 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.230660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.230716 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.230818 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:46.231050 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.231536 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.232470 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.232697 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.232808 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.233119 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:46.233479 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:01:46.233639 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.233727 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.233763 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:01:46.239355 25896 raft_consensus.cc:1261] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 1 FOLLOWER]: Refusing update from remote peer 31c261dc16ad486c93869483aafc82f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:46.239356 25762 raft_consensus.cc:1261] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 1 FOLLOWER]: Refusing update from remote peer 31c261dc16ad486c93869483aafc82f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:46.240015 25970 consensus_queue.cc:1035] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6d1f7d25844f97b257ce5703cf6a3d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 44153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:46.240227 25966 consensus_queue.cc:1035] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1392e4d0aa074c029b5c6a66779726eb" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 35411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:46.245935 25973 mvcc.cc:205] Tried to move back new op lower bound from 7298939724749815808 to 7298939724683526144. Current Snapshot: MvccSnapshot[applied={T|T < 7298939724749815808}]
I20260620 15:01:46.249336 25974 mvcc.cc:205] Tried to move back new op lower bound from 7298939724749815808 to 7298939724683526144. Current Snapshot: MvccSnapshot[applied={T|T < 7298939724749815808}]
I20260620 15:01:46.249356 25975 mvcc.cc:205] Tried to move back new op lower bound from 7298939724749815808 to 7298939724683526144. Current Snapshot: MvccSnapshot[applied={T|T < 7298939724749815808 or (T in {7298939724749815808})}]
I20260620 15:01:46.256706 25481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:46.256886 25481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:46.259166 25481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54a3e42d3af6407aa28cba19264d66b5: Sending DeleteTablet for 3 replicas of tablet 5cc7685f7bd5478a88f3f94813f0806f
I20260620 15:01:46.259922 25742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc7685f7bd5478a88f3f94813f0806f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:46 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:34524
I20260620 15:01:46.259969 25876 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc7685f7bd5478a88f3f94813f0806f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:46 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:59272
I20260620 15:01:46.260185 25608 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc7685f7bd5478a88f3f94813f0806f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:46 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:46850
I20260620 15:01:46.260370 25983 tablet_replica.cc:330] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: stopping tablet replica
I20260620 15:01:46.260576 25985 tablet_replica.cc:330] T 5cc7685f7bd5478a88f3f94813f0806f P 31c261dc16ad486c93869483aafc82f1: stopping tablet replica
I20260620 15:01:46.260640 25983 raft_consensus.cc:2229] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:46.260958 25983 raft_consensus.cc:2258] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:46.261166 25984 tablet_replica.cc:330] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: stopping tablet replica
I20260620 15:01:46.261473 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 25545
I20260620 15:01:46.262334 25983 ts_tablet_manager.cc:1905] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:46.262749 25984 raft_consensus.cc:2229] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:46.263089 25984 raft_consensus.cc:2258] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:46.264418 25984 ts_tablet_manager.cc:1905] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:46.265719 25983 ts_tablet_manager.cc:1918] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:46.265818 25983 log.cc:1196] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/5cc7685f7bd5478a88f3f94813f0806f
I20260620 15:01:46.266178 25983 ts_tablet_manager.cc:1939] T 5cc7685f7bd5478a88f3f94813f0806f P ce6d1f7d25844f97b257ce5703cf6a3d: Deleting consensus metadata
I20260620 15:01:46.266516 25984 ts_tablet_manager.cc:1918] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:46.266618 25984 log.cc:1196] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/5cc7685f7bd5478a88f3f94813f0806f
I20260620 15:01:46.266899 25984 ts_tablet_manager.cc:1939] T 5cc7685f7bd5478a88f3f94813f0806f P 1392e4d0aa074c029b5c6a66779726eb: Deleting consensus metadata
I20260620 15:01:46.267467 25469 catalog_manager.cc:4841] TS ce6d1f7d25844f97b257ce5703cf6a3d (127.7.46.130:44153): tablet 5cc7685f7bd5478a88f3f94813f0806f (table test-table [id=273f9d6caf1743d5b7d98caf2c0adf6e]) successfully deleted
I20260620 15:01:46.268007 25467 catalog_manager.cc:4841] TS 1392e4d0aa074c029b5c6a66779726eb (127.7.46.131:35411): tablet 5cc7685f7bd5478a88f3f94813f0806f (table test-table [id=273f9d6caf1743d5b7d98caf2c0adf6e]) successfully deleted
W20260620 15:01:46.269292 25470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:34645 (error 108)
I20260620 15:01:46.269589 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 25679
I20260620 15:01:46.270323 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.270082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:34645 (local address 127.0.0.1:46862)
0620 15:01:46.270182 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.270244 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:34645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 15:01:46.270491 25470 catalog_manager.cc:4584] TS 31c261dc16ad486c93869483aafc82f1 (127.7.46.129:34645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cc7685f7bd5478a88f3f94813f0806f: Network error: Client connection negotiation failed: client connection to 127.7.46.129:34645: connect: Connection refused (error 111)
I20260620 15:01:46.275885 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 25813
I20260620 15:01:46.281651 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 25451
2026-06-20T15:01:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2246 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T15:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:46Z Disabled control of system clock
I20260620 15:01:46.312146 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37615
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:37615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:01:46.417918 25990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:46.418169 25990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:46.418205 25990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:46.422170 25990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:46.422221 25990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:46.422237 25990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:46.422253 25990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:46.422268 25990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:46.426875 25990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:37615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:46.427816 25990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:46.429088 25990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:46.437075 25998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:46.437093 25995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:46.437126 25996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:46.437256 25990 server_base.cc:1034] running on GCE node
I20260620 15:01:46.437942 25990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:46.438899 25990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:46.440056 25990 hybrid_clock.cc:648] HybridClock initialized: now 1781967706440024 us; error 44 us; skew 500 ppm
I20260620 15:01:46.441764 25990 webserver.cc:460] Webserver started at http://127.7.46.190:34619/ using document root <none> and password file <none>
I20260620 15:01:46.442315 25990 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:46.442373 25990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:46.442615 25990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:46.444605 25990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "b692ee33224b49ad8a86863e2e37db1e"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "d993301f30cfdf9197e6b11f267e8d19"
server_key_iv: "a5c0b099d51f42ba1d3d85a89d78b2ac"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.445159 25990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "b692ee33224b49ad8a86863e2e37db1e"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "d993301f30cfdf9197e6b11f267e8d19"
server_key_iv: "a5c0b099d51f42ba1d3d85a89d78b2ac"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.448721 25990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:01:46.451272 26004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.452486 25990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:01:46.452615 25990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "b692ee33224b49ad8a86863e2e37db1e"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "d993301f30cfdf9197e6b11f267e8d19"
server_key_iv: "a5c0b099d51f42ba1d3d85a89d78b2ac"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.452718 25990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:46.462188 25990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:46.462787 25990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:46.462975 25990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:46.469689 26056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:37615 every 8 connection(s)
I20260620 15:01:46.469738 25990 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:37615
I20260620 15:01:46.470628 25990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:46.473652 26057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:46.477257 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 25990
I20260620 15:01:46.477401 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:46.477726 7354 external_mini_cluster.cc:1428] Setting key f3b91a351ae5f5bbbdcc9b350c54a733
I20260620 15:01:46.479568 26057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e: Bootstrap starting.
I20260620 15:01:46.481721 26057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:46.482441 26057 log.cc:824] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:46.484280 26057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e: No bootstrap required, opened a new log
I20260620 15:01:46.486773 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.479093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.7.46.190:37615)
0620 15:01:46.479712 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.479727 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.479761 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:01:46.480430 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.480446 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.480815 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.481133 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:46.481554 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.482597 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.483453 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.483980 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.484159 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.485000 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:46.485005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:46.485017 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:46.485389 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:46.485441 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:46.485460 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:46.485568 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.485681 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.485933 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260620 15:01:46.487304 26057 raft_consensus.cc:348] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692ee33224b49ad8a86863e2e37db1e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37615 } }
I20260620 15:01:46.487541 26057 raft_consensus.cc:374] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:46.487610 26057 raft_consensus.cc:729] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b692ee33224b49ad8a86863e2e37db1e, State: Initialized, Role: FOLLOWER
I20260620 15:01:46.488096 26057 consensus_queue.cc:260] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692ee33224b49ad8a86863e2e37db1e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37615 } }
I20260620 15:01:46.488252 26057 raft_consensus.cc:388] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:46.488327 26057 raft_consensus.cc:482] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:46.488443 26057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:46.489415 26057 raft_consensus.cc:504] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692ee33224b49ad8a86863e2e37db1e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37615 } }
I20260620 15:01:46.489737 26057 leader_election.cc:302] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b692ee33224b49ad8a86863e2e37db1e; no voters:
I20260620 15:01:46.490020 26057 leader_election.cc:288] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:46.490151 26062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:46.490383 26062 raft_consensus.cc:686] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [term 1 LEADER]: Becoming Leader. State: Replica: b692ee33224b49ad8a86863e2e37db1e, State: Running, Role: LEADER
I20260620 15:01:46.490656 26062 consensus_queue.cc:237] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692ee33224b49ad8a86863e2e37db1e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37615 } }
I20260620 15:01:46.491178 26057 sys_catalog.cc:564] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:46.492240 26063 sys_catalog.cc:455] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b692ee33224b49ad8a86863e2e37db1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692ee33224b49ad8a86863e2e37db1e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37615 } } }
I20260620 15:01:46.492372 26063 sys_catalog.cc:458] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:46.492686 26064 sys_catalog.cc:455] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b692ee33224b49ad8a86863e2e37db1e. Latest consensus state: current_term: 1 leader_uuid: "b692ee33224b49ad8a86863e2e37db1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692ee33224b49ad8a86863e2e37db1e" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37615 } } }
I20260620 15:01:46.492753 26071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:46.492810 26064 sys_catalog.cc:458] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:46.496322 26071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:46.501356 26071 catalog_manager.cc:1347] Generated new cluster ID: 4b117c54930148fd91216874b42e0218
I20260620 15:01:46.501438 26071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:46.514613 26071 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:46.515589 26071 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:46.520421 26071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e: Generated new TSK 0
I20260620 15:01:46.520998 26071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:46.524626 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37615
--builtin_ntp_servers=127.7.46.148:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:01:46.629731 26081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:46.629972 26081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:46.630010 26081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:46.634048 26081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:46.634099 26081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:46.634183 26081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:01:46.638809 26081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:46.639737 26081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:46.641059 26081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:46.650179 26089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:46.650203 26086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:46.650347 26081 server_base.cc:1034] running on GCE node
W20260620 15:01:46.650204 26087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:46.650868 26081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:46.651445 26081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:46.652599 26081 hybrid_clock.cc:648] HybridClock initialized: now 1781967706652583 us; error 28 us; skew 500 ppm
I20260620 15:01:46.654172 26081 webserver.cc:460] Webserver started at http://127.7.46.129:45215/ using document root <none> and password file <none>
I20260620 15:01:46.654670 26081 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:46.654726 26081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:46.654872 26081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:46.656935 26081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "227f6c7d28d847b7bf2d8d200cfde67d"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "f8fa5a3ff714ea45d4239e45ca9ba414"
server_key_iv: "f1dc7e2bbb5b89e1dffb898d2ac098ba"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.657408 26081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "227f6c7d28d847b7bf2d8d200cfde67d"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "f8fa5a3ff714ea45d4239e45ca9ba414"
server_key_iv: "f1dc7e2bbb5b89e1dffb898d2ac098ba"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.660712 26081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:46.662961 26095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.664119 26081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:46.664206 26081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "227f6c7d28d847b7bf2d8d200cfde67d"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "f8fa5a3ff714ea45d4239e45ca9ba414"
server_key_iv: "f1dc7e2bbb5b89e1dffb898d2ac098ba"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.664283 26081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:46.684958 26081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:46.685510 26081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:46.685642 26081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:46.686117 26081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:46.686949 26081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:46.686995 26081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.687033 26081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:46.687048 26081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.695438 26081 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:37055
I20260620 15:01:46.695510 26208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:37055 every 8 connection(s)
I20260620 15:01:46.696318 26081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:46.700532 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 26081
I20260620 15:01:46.700659 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:46.700969 7354 external_mini_cluster.cc:1428] Setting key d2d07015dd3ec06ffe09b46fe0b18e3e
I20260620 15:01:46.703001 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37615
--builtin_ntp_servers=127.7.46.148:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:01:46.706462 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.698092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:47373 (local address 127.7.46.190:37615)
0620 15:01:46.698228 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.698232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.699211 (+ 979us) server_negotiation.cc:407] Connection header received
0620 15:01:46.700542 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.700546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.700606 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.700717 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:46.701810 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.702546 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.704297 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.704697 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.704817 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.705883 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:46.705891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:46.705897 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:46.705914 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:46.705941 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:46.705955 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:46.706034 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.706176 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.706245 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:01:46.707165 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.698480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37615 (local address 127.7.46.129:47373)
0620 15:01:46.699081 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.699118 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:46.700333 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:46.700910 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:46.700920 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:46.701027 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:46.701650 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.701671 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.702672 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.702676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.704161 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.704175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.704844 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.704847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.704985 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.705630 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:46.705631 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:46.705668 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:46.705726 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:46.706040 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:46.706046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:46.706050 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 15:01:46.706138 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 15:01:46.706334 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260620 15:01:46.708374 26209 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37615
I20260620 15:01:46.708581 26209 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:46.709053 26209 heartbeater.cc:511] Master 127.7.46.190:37615 requested a full tablet report, sending...
I20260620 15:01:46.710423 26021 ts_manager.cc:194] Registered new tserver with Master: 227f6c7d28d847b7bf2d8d200cfde67d (127.7.46.129:37055)
I20260620 15:01:46.711509 26021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.129:47373
W20260620 15:01:46.815697 26212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:46.815996 26212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:46.816063 26212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:46.820012 26212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:46.820056 26212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:46.820145 26212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:01:46.824757 26212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:46.825707 26212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:46.827005 26212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:46.836467 26218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:46.836550 26212 server_base.cc:1034] running on GCE node
W20260620 15:01:46.836524 26217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:46.836714 26220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:46.837179 26212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:46.837778 26212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:46.838929 26212 hybrid_clock.cc:648] HybridClock initialized: now 1781967706838899 us; error 44 us; skew 500 ppm
I20260620 15:01:46.840732 26212 webserver.cc:460] Webserver started at http://127.7.46.130:45339/ using document root <none> and password file <none>
I20260620 15:01:46.841285 26212 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:46.841344 26212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:46.841589 26212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:46.843503 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "e602bcfb1a004dc197e657b08d17f09d"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "cb2cdcc437f48dd471c0d75146c0ef62"
server_key_iv: "92f369ac0df78f5db12d4c666d0b3cac"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.844094 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "e602bcfb1a004dc197e657b08d17f09d"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "cb2cdcc437f48dd471c0d75146c0ef62"
server_key_iv: "92f369ac0df78f5db12d4c666d0b3cac"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.847576 26212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:46.849967 26226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.851110 26212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:46.851238 26212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "e602bcfb1a004dc197e657b08d17f09d"
format_stamp: "Formatted at 2026-06-20 15:01:46 on dist-test-slave-30t4"
server_key: "cb2cdcc437f48dd471c0d75146c0ef62"
server_key_iv: "92f369ac0df78f5db12d4c666d0b3cac"
server_key_version: "encryptionkey@0"
I20260620 15:01:46.851344 26212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:46.868216 26212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:46.868808 26212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:46.869011 26212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:46.869547 26212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:46.870435 26212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:46.870507 26212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.870572 26212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:46.870625 26212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:46.879190 26212 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:35487
I20260620 15:01:46.879213 26339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:35487 every 8 connection(s)
I20260620 15:01:46.880064 26212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:46.889036 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.881800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:45077 (local address 127.7.46.190:37615)
0620 15:01:46.881942 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:46.881946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:46.882964 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 15:01:46.884386 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:46.884389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:46.884435 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:46.884512 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:46.885319 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.885834 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.887178 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.887397 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.887515 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.888558 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:46.888562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:46.888565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:46.888583 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:46.888601 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:46.888608 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:46.888681 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:46.888838 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 15:01:46.888897 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:01:46.889762 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 26212
I20260620 15:01:46.889871 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:46.889950 26342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:46.882222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37615 (local address 127.7.46.130:45077)
0620 15:01:46.882828 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:46.882866 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:46.884163 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:46.884663 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:46.884669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:46.884746 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:46.885197 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.885209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.885970 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.885973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.887064 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:46.887074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:46.887539 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:46.887544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:46.887718 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:46.888313 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:46.888314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:46.888351 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:46.888407 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:46.888694 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:46.888703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:46.888709 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:01:46.888838 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 15:01:46.889105 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260620 15:01:46.890209 7354 external_mini_cluster.cc:1428] Setting key e106f6ee1ddea7fe5beafd7b6ceac548
I20260620 15:01:46.891047 26340 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37615
I20260620 15:01:46.891275 26340 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:46.891884 26340 heartbeater.cc:511] Master 127.7.46.190:37615 requested a full tablet report, sending...
I20260620 15:01:46.892372 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37615
--builtin_ntp_servers=127.7.46.148:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:01:46.893165 26021 ts_manager.cc:194] Registered new tserver with Master: e602bcfb1a004dc197e657b08d17f09d (127.7.46.130:35487)
I20260620 15:01:46.893749 26021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.130:45077
W20260620 15:01:46.998860 26343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:46.999083 26343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:46.999146 26343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:47.003069 26343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:47.003115 26343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:47.003203 26343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:01:47.009136 26343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:37615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:47.010200 26343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:47.011483 26343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:47.020610 26348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:47.020685 26351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:47.020582 26349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:47.020789 26343 server_base.cc:1034] running on GCE node
I20260620 15:01:47.021294 26343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:47.021915 26343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:47.023092 26343 hybrid_clock.cc:648] HybridClock initialized: now 1781967707023056 us; error 49 us; skew 500 ppm
I20260620 15:01:47.024760 26343 webserver.cc:460] Webserver started at http://127.7.46.131:35969/ using document root <none> and password file <none>
I20260620 15:01:47.025303 26343 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:47.025393 26343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:47.025617 26343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:47.027532 26343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "26964b40ed824837a78b6eb661134a5f"
format_stamp: "Formatted at 2026-06-20 15:01:47 on dist-test-slave-30t4"
server_key: "cc0ae85375fd261a3753b9a8bf3abddb"
server_key_iv: "c559c4d7e4e8c3118d38737e10f648f3"
server_key_version: "encryptionkey@0"
I20260620 15:01:47.028085 26343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "26964b40ed824837a78b6eb661134a5f"
format_stamp: "Formatted at 2026-06-20 15:01:47 on dist-test-slave-30t4"
server_key: "cc0ae85375fd261a3753b9a8bf3abddb"
server_key_iv: "c559c4d7e4e8c3118d38737e10f648f3"
server_key_version: "encryptionkey@0"
I20260620 15:01:47.031387 26343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:01:47.033679 26357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:47.034875 26343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:01:47.035010 26343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "26964b40ed824837a78b6eb661134a5f"
format_stamp: "Formatted at 2026-06-20 15:01:47 on dist-test-slave-30t4"
server_key: "cc0ae85375fd261a3753b9a8bf3abddb"
server_key_iv: "c559c4d7e4e8c3118d38737e10f648f3"
server_key_version: "encryptionkey@0"
I20260620 15:01:47.035113 26343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:47.060424 26343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:47.061025 26343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:47.061218 26343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:47.061766 26343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:47.062629 26343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:47.062697 26343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:47.062767 26343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:47.062819 26343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:47.071844 26343 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:43055
I20260620 15:01:47.071890 26470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:43055 every 8 connection(s)
I20260620 15:01:47.072722 26343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:47.078938 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 26343
I20260620 15:01:47.079037 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:47.079316 7354 external_mini_cluster.cc:1428] Setting key e620c2795fd70c301d799382951097f1
I20260620 15:01:47.081812 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.074493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:55131 (local address 127.7.46.190:37615)
0620 15:01:47.074656 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:47.074661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:47.075639 (+ 978us) server_negotiation.cc:407] Connection header received
0620 15:01:47.076954 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:47.076957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:47.077007 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:47.077078 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:47.077862 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.078367 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.079855 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.080147 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.080259 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.081328 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:47.081332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:47.081335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:47.081347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:47.081373 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:47.081382 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:47.081454 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:47.081622 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 15:01:47.081678 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:01:47.082576 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.074877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:37615 (local address 127.7.46.131:55131)
0620 15:01:47.075490 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:47.075527 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:47.076772 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:47.077232 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:47.077238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:47.077315 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:47.077752 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.077763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.078468 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.078470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.079687 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.079696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.080288 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.080293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.080466 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.081045 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:47.081047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:47.081096 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:47.081170 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:47.081472 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:47.081487 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:47.081493 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:01:47.081619 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 15:01:47.081867 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260620 15:01:47.083546 26471 heartbeater.cc:347] Connected to a master server at 127.7.46.190:37615
I20260620 15:01:47.083804 26471 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:47.084273 26471 heartbeater.cc:511] Master 127.7.46.190:37615 requested a full tablet report, sending...
I20260620 15:01:47.085291 26021 ts_manager.cc:194] Registered new tserver with Master: 26964b40ed824837a78b6eb661134a5f (127.7.46.131:43055)
I20260620 15:01:47.085822 26021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.131:55131
I20260620 15:01:47.093114 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:01:47.098626 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.094939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.7.46.190:37615)
0620 15:01:47.095100 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:47.095103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:47.095188 (+ 85us) server_negotiation.cc:407] Connection header received
0620 15:01:47.095280 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:47.095282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:47.095316 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:47.095378 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:47.095636 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.096130 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.096920 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.097118 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.097193 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.098071 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:47.098074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:47.098076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:47.098085 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:47.098100 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:47.098108 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:47.098199 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:47.098320 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 15:01:47.098473 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 15:01:47.101518 26021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:01:47.104326 26021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:01:47.118249 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.111950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:43055 (local address 127.0.0.1:57168)
0620 15:01:47.113005 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:47.113018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:47.113092 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:47.113719 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:47.113725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:47.113746 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:47.113868 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.113879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.115316 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.115322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.116731 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.116739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.117648 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.117653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.117779 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.117826 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 15:01:47.117962 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:01:47.118089 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":152,"thread_start_us":100,"threads_started":1}
I20260620 15:01:47.118285 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.112161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.7.46.131:43055)
0620 15:01:47.113065 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:47.113077 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:47.113101 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:01:47.113185 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:47.113191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:47.113395 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:47.113559 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:47.114002 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.115185 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.116848 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.117517 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.117623 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.117800 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:47.118011 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 15:01:47.118119 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":105,"threads_started":1}
I20260620 15:01:47.118574 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.111888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:37055 (local address 127.0.0.1:51742)
0620 15:01:47.112625 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:47.112671 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:47.112793 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:47.113974 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:47.113990 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:47.114009 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:47.114133 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.114139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.115398 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.115401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.117062 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.117074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.118108 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.118111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.118221 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.118244 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:01:47.118305 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 15:01:47.118405 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":165,"thread_start_us":73,"threads_started":1}
I20260620 15:01:47.118783 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.112633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:35487 (local address 127.0.0.1:36754)
0620 15:01:47.113298 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:47.113311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:47.113419 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:47.113962 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:47.113966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:47.113985 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:47.114259 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.114263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.115271 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.115276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.117062 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.117074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.118297 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.118304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.118470 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.118493 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:01:47.118571 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 15:01:47.118640 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":277,"thread_start_us":96,"threads_started":1}
I20260620 15:01:47.118896 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.112802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.7.46.130:35487)
0620 15:01:47.113277 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:47.113290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:47.113534 (+ 244us) server_negotiation.cc:407] Connection header received
0620 15:01:47.113623 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:47.113631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:47.113809 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:47.113990 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:47.114367 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.115146 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.117221 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.117955 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.118053 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.118226 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:47.118626 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 15:01:47.118727 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":139,"threads_started":1}
I20260620 15:01:47.119494 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.111967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.7.46.129:37055)
0620 15:01:47.113529 (+ 1562us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:47.113541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:47.113563 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:01:47.113634 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:47.113639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:47.113819 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:47.114089 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:47.114221 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.115288 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.117195 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.117974 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.118673 (+ 699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.119154 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:47.119212 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 15:01:47.119312 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448,"thread_start_us":111,"threads_started":1}
I20260620 15:01:47.121928 26143 tablet_service.cc:1467] Processing CreateTablet for tablet f7fba7d147e04918bea9c304ad0ed3c1 (DEFAULT_TABLE table=test-table [id=619b58cd96624bf5a983ba85006f52ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:47.122025 26274 tablet_service.cc:1467] Processing CreateTablet for tablet f7fba7d147e04918bea9c304ad0ed3c1 (DEFAULT_TABLE table=test-table [id=619b58cd96624bf5a983ba85006f52ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:47.122026 26405 tablet_service.cc:1467] Processing CreateTablet for tablet f7fba7d147e04918bea9c304ad0ed3c1 (DEFAULT_TABLE table=test-table [id=619b58cd96624bf5a983ba85006f52ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:47.122901 26143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7fba7d147e04918bea9c304ad0ed3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:47.123071 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7fba7d147e04918bea9c304ad0ed3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:47.123071 26274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7fba7d147e04918bea9c304ad0ed3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:47.129387 26488 tablet_bootstrap.cc:492] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d: Bootstrap starting.
I20260620 15:01:47.130363 26489 tablet_bootstrap.cc:492] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: Bootstrap starting.
I20260620 15:01:47.131675 26488 tablet_bootstrap.cc:654] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:47.132400 26489 tablet_bootstrap.cc:654] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:47.132555 26488 log.cc:824] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:47.133217 26489 log.cc:824] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:47.133256 26490 tablet_bootstrap.cc:492] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: Bootstrap starting.
I20260620 15:01:47.135305 26490 tablet_bootstrap.cc:654] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:47.135354 26489 tablet_bootstrap.cc:492] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: No bootstrap required, opened a new log
I20260620 15:01:47.135567 26489 ts_tablet_manager.cc:1397] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:01:47.136000 26488 tablet_bootstrap.cc:492] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d: No bootstrap required, opened a new log
I20260620 15:01:47.136103 26490 log.cc:824] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:47.136214 26488 ts_tablet_manager.cc:1397] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 15:01:47.137596 26490 tablet_bootstrap.cc:492] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: No bootstrap required, opened a new log
I20260620 15:01:47.137781 26490 ts_tablet_manager.cc:1397] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:01:47.140300 26489 raft_consensus.cc:348] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.140588 26489 raft_consensus.cc:374] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:47.140651 26489 raft_consensus.cc:729] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26964b40ed824837a78b6eb661134a5f, State: Initialized, Role: FOLLOWER
I20260620 15:01:47.140748 26488 raft_consensus.cc:348] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.141000 26488 raft_consensus.cc:374] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:47.141050 26488 raft_consensus.cc:729] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227f6c7d28d847b7bf2d8d200cfde67d, State: Initialized, Role: FOLLOWER
I20260620 15:01:47.141160 26489 consensus_queue.cc:260] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.141103 26490 raft_consensus.cc:348] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.141333 26490 raft_consensus.cc:374] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:47.141384 26490 raft_consensus.cc:729] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e602bcfb1a004dc197e657b08d17f09d, State: Initialized, Role: FOLLOWER
I20260620 15:01:47.141513 26488 consensus_queue.cc:260] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.141749 26490 consensus_queue.cc:260] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.142182 26489 ts_tablet_manager.cc:1428] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 15:01:47.142275 26209 heartbeater.cc:503] Master 127.7.46.190:37615 was elected leader, sending a full tablet report...
I20260620 15:01:47.142531 26471 heartbeater.cc:503] Master 127.7.46.190:37615 was elected leader, sending a full tablet report...
I20260620 15:01:47.142550 26488 ts_tablet_manager.cc:1428] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 15:01:47.143039 26490 ts_tablet_manager.cc:1428] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 15:01:47.143369 26340 heartbeater.cc:503] Master 127.7.46.190:37615 was elected leader, sending a full tablet report...
W20260620 15:01:47.197458 26210 tablet.cc:2369] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:47.210356 26494 raft_consensus.cc:482] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:47.210525 26494 raft_consensus.cc:504] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.211615 26494 leader_election.cc:288] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227f6c7d28d847b7bf2d8d200cfde67d (127.7.46.129:37055), e602bcfb1a004dc197e657b08d17f09d (127.7.46.130:35487)
I20260620 15:01:47.214982 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.211788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:37055 (local address 127.7.46.131:39753)
0620 15:01:47.211930 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:47.211946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:47.212034 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:47.212303 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:47.212306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:47.212323 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:47.212438 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.212444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.213179 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.213182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.213950 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.213958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.214577 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.214580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.214675 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.214698 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:01:47.214789 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 15:01:47.214858 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:01:47.214991 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.211882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:39753 (local address 127.7.46.129:37055)
0620 15:01:47.212018 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:47.212025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:47.212046 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:01:47.212130 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:47.212133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:47.212188 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:47.212289 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:47.212563 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.213066 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.214081 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.214455 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.214577 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.214626 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:47.214802 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 15:01:47.214864 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:01:47.215345 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.211989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:32875 (local address 127.7.46.130:35487)
0620 15:01:47.212115 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:47.212120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:47.212368 (+ 248us) server_negotiation.cc:407] Connection header received
0620 15:01:47.212513 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:47.212518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:47.212573 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:47.212656 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:47.212917 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.213398 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.214386 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.214763 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.214891 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.214935 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:47.215167 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 15:01:47.215235 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:01:47.215585 26163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fba7d147e04918bea9c304ad0ed3c1" candidate_uuid: "26964b40ed824837a78b6eb661134a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" is_pre_election: true
I20260620 15:01:47.215674 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.211953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:35487 (local address 127.7.46.131:32875)
0620 15:01:47.212281 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:47.212292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:47.212403 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:47.212666 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:47.212669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:47.212683 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:47.212804 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.212815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.213509 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.213512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.214265 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:47.214273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.214884 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.214888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:47.214981 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.215028 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 15:01:47.215468 (+ 440us) client_negotiation.cc:240] Negotiation successful
0620 15:01:47.215536 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
I20260620 15:01:47.215909 26163 raft_consensus.cc:2454] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26964b40ed824837a78b6eb661134a5f in term 0.
I20260620 15:01:47.216372 26294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fba7d147e04918bea9c304ad0ed3c1" candidate_uuid: "26964b40ed824837a78b6eb661134a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e602bcfb1a004dc197e657b08d17f09d" is_pre_election: true
I20260620 15:01:47.216392 26361 leader_election.cc:302] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227f6c7d28d847b7bf2d8d200cfde67d, 26964b40ed824837a78b6eb661134a5f; no voters:
I20260620 15:01:47.216662 26294 raft_consensus.cc:2454] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26964b40ed824837a78b6eb661134a5f in term 0.
I20260620 15:01:47.216696 26494 raft_consensus.cc:2781] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:01:47.216779 26494 raft_consensus.cc:482] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:01:47.216831 26494 raft_consensus.cc:3037] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:47.217931 26494 raft_consensus.cc:504] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.218245 26494 leader_election.cc:288] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [CANDIDATE]: Term 1 election: Requested vote from peers 227f6c7d28d847b7bf2d8d200cfde67d (127.7.46.129:37055), e602bcfb1a004dc197e657b08d17f09d (127.7.46.130:35487)
I20260620 15:01:47.218520 26163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fba7d147e04918bea9c304ad0ed3c1" candidate_uuid: "26964b40ed824837a78b6eb661134a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227f6c7d28d847b7bf2d8d200cfde67d"
I20260620 15:01:47.218580 26294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fba7d147e04918bea9c304ad0ed3c1" candidate_uuid: "26964b40ed824837a78b6eb661134a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e602bcfb1a004dc197e657b08d17f09d"
I20260620 15:01:47.218645 26163 raft_consensus.cc:3037] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:47.218706 26294 raft_consensus.cc:3037] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:47.219589 26163 raft_consensus.cc:2454] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26964b40ed824837a78b6eb661134a5f in term 1.
I20260620 15:01:47.219715 26294 raft_consensus.cc:2454] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26964b40ed824837a78b6eb661134a5f in term 1.
I20260620 15:01:47.219878 26361 leader_election.cc:302] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227f6c7d28d847b7bf2d8d200cfde67d, 26964b40ed824837a78b6eb661134a5f; no voters:
I20260620 15:01:47.220063 26494 raft_consensus.cc:2781] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:47.220275 26494 raft_consensus.cc:686] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 1 LEADER]: Becoming Leader. State: Replica: 26964b40ed824837a78b6eb661134a5f, State: Running, Role: LEADER
I20260620 15:01:47.220593 26494 consensus_queue.cc:237] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } }
I20260620 15:01:47.223541 26020 catalog_manager.cc:5495] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 26964b40ed824837a78b6eb661134a5f (127.7.46.131). New cstate: current_term: 1 leader_uuid: "26964b40ed824837a78b6eb661134a5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26964b40ed824837a78b6eb661134a5f" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 } health_report { overall_health: UNKNOWN } } }
I20260620 15:01:47.239030 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.235380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.7.46.131:43055)
0620 15:01:47.235529 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:47.235535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:47.235551 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:01:47.235731 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:47.235734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:47.235832 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:47.235961 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:47.236199 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.236682 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.237545 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:47.237770 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:47.237879 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:47.238193 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:47.238602 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:01:47.238753 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:47.238851 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 15:01:47.238884 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:01:47.244361 26163 raft_consensus.cc:1261] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 1 FOLLOWER]: Refusing update from remote peer 26964b40ed824837a78b6eb661134a5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:47.244364 26294 raft_consensus.cc:1261] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 1 FOLLOWER]: Refusing update from remote peer 26964b40ed824837a78b6eb661134a5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:47.245026 26494 consensus_queue.cc:1035] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "227f6c7d28d847b7bf2d8d200cfde67d" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:47.245235 26498 consensus_queue.cc:1035] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e602bcfb1a004dc197e657b08d17f09d" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:47.253356 26502 mvcc.cc:205] Tried to move back new op lower bound from 7298939728866537472 to 7298939728777830400. Current Snapshot: MvccSnapshot[applied={T|T < 7298939728866537472}]
I20260620 15:01:47.255097 26503 mvcc.cc:205] Tried to move back new op lower bound from 7298939728866537472 to 7298939728777830400. Current Snapshot: MvccSnapshot[applied={T|T < 7298939728866537472}]
I20260620 15:01:47.261318 26020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:47.261471 26020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:47.263618 26020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b692ee33224b49ad8a86863e2e37db1e: Sending DeleteTablet for 3 replicas of tablet f7fba7d147e04918bea9c304ad0ed3c1
I20260620 15:01:47.264377 26405 tablet_service.cc:1514] Processing DeleteTablet for tablet f7fba7d147e04918bea9c304ad0ed3c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:47 UTC) from {username='slave'} at 127.0.0.1:57168
I20260620 15:01:47.264374 26143 tablet_service.cc:1514] Processing DeleteTablet for tablet f7fba7d147e04918bea9c304ad0ed3c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:47 UTC) from {username='slave'} at 127.0.0.1:51742
I20260620 15:01:47.264460 26274 tablet_service.cc:1514] Processing DeleteTablet for tablet f7fba7d147e04918bea9c304ad0ed3c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:47 UTC) from {username='slave'} at 127.0.0.1:36754
I20260620 15:01:47.264887 26512 tablet_replica.cc:330] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: stopping tablet replica
I20260620 15:01:47.265079 26513 tablet_replica.cc:330] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d: stopping tablet replica
I20260620 15:01:47.265187 26512 raft_consensus.cc:2229] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:47.265290 26513 raft_consensus.cc:2229] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:47.265452 26512 raft_consensus.cc:2258] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:47.265511 26513 raft_consensus.cc:2258] T f7fba7d147e04918bea9c304ad0ed3c1 P 227f6c7d28d847b7bf2d8d200cfde67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:47.265718 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 26081
I20260620 15:01:47.267100 26511 tablet_replica.cc:330] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: stopping tablet replica
I20260620 15:01:47.267340 26511 raft_consensus.cc:2229] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:01:47.267683 26511 raft_consensus.cc:2258] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:47.267760 26512 ts_tablet_manager.cc:1905] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:47.268855 26511 ts_tablet_manager.cc:1905] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:47.270370 26512 ts_tablet_manager.cc:1918] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:47.270465 26512 log.cc:1196] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/f7fba7d147e04918bea9c304ad0ed3c1
I20260620 15:01:47.270774 26512 ts_tablet_manager.cc:1939] T f7fba7d147e04918bea9c304ad0ed3c1 P e602bcfb1a004dc197e657b08d17f09d: Deleting consensus metadata
I20260620 15:01:47.271497 26511 ts_tablet_manager.cc:1918] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:47.271586 26511 log.cc:1196] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/f7fba7d147e04918bea9c304ad0ed3c1
I20260620 15:01:47.271919 26511 ts_tablet_manager.cc:1939] T f7fba7d147e04918bea9c304ad0ed3c1 P 26964b40ed824837a78b6eb661134a5f: Deleting consensus metadata
I20260620 15:01:47.271950 26006 catalog_manager.cc:4841] TS e602bcfb1a004dc197e657b08d17f09d (127.7.46.130:35487): tablet f7fba7d147e04918bea9c304ad0ed3c1 (table test-table [id=619b58cd96624bf5a983ba85006f52ff]) successfully deleted
I20260620 15:01:47.272833 26007 catalog_manager.cc:4841] TS 26964b40ed824837a78b6eb661134a5f (127.7.46.131:43055): tablet f7fba7d147e04918bea9c304ad0ed3c1 (table test-table [id=619b58cd96624bf5a983ba85006f52ff]) successfully deleted
W20260620 15:01:47.274260 26008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:37055 (error 108)
I20260620 15:01:47.274448 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 26212
I20260620 15:01:47.276537 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:47.276284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:37055 (local address 127.0.0.1:51754)
0620 15:01:47.276406 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:47.276468 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:37055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:01:47.276738 26008 catalog_manager.cc:4584] TS 227f6c7d28d847b7bf2d8d200cfde67d (127.7.46.129:37055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7fba7d147e04918bea9c304ad0ed3c1: Network error: Client connection negotiation failed: client connection to 127.7.46.129:37055: connect: Connection refused (error 111)
I20260620 15:01:47.281359 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 26343
I20260620 15:01:47.286300 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 25990
2026-06-20T15:01:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1002 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:47 dist-test-slave-30t4 krb5kdc[26517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:47 dist-test-slave-30t4 krb5kdc[26517](info): set up 2 sockets
Jun 20 15:01:47 dist-test-slave-30t4 krb5kdc[26517](info): commencing operation
krb5kdc: starting...
W20260620 15:01:48.311748 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:48 dist-test-slave-30t4 krb5kdc[26517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:01:48.430433 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:42685
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:48.537045 26530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:48.537318 26530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:48.537397 26530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:48.541342 26530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:48.541421 26530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:48.541460 26530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:48.541519 26530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:48.541558 26530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:48.546593 26530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:48.547559 26530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:48.548527 26530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:48.556443 26536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:48.556453 26535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:48.556552 26530 server_base.cc:1034] running on GCE node
W20260620 15:01:48.556720 26538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:48.557196 26530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:48.558197 26530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:48.559381 26530 hybrid_clock.cc:648] HybridClock initialized: now 1781967708559355 us; error 37 us; skew 500 ppm
Jun 20 15:01:48 dist-test-slave-30t4 krb5kdc[26517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967708, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:48.561987 26530 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:01:48.562860 26530 webserver.cc:460] Webserver started at http://127.7.46.190:36003/ using document root <none> and password file <none>
I20260620 15:01:48.563388 26530 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:48.563483 26530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:48.563700 26530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:48.565644 26530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "342b7079765c4ab1a2e347ebe0a9b57b"
format_stamp: "Formatted at 2026-06-20 15:01:48 on dist-test-slave-30t4"
server_key: "603cd454699925701e97e2480ddb247b"
server_key_iv: "ac59d14ca70860172b8ef075ee17b326"
server_key_version: "encryptionkey@0"
I20260620 15:01:48.566188 26530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "342b7079765c4ab1a2e347ebe0a9b57b"
format_stamp: "Formatted at 2026-06-20 15:01:48 on dist-test-slave-30t4"
server_key: "603cd454699925701e97e2480ddb247b"
server_key_iv: "ac59d14ca70860172b8ef075ee17b326"
server_key_version: "encryptionkey@0"
I20260620 15:01:48.569653 26530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:01:48.571902 26545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:48.573040 26530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:01:48.573171 26530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "342b7079765c4ab1a2e347ebe0a9b57b"
format_stamp: "Formatted at 2026-06-20 15:01:48 on dist-test-slave-30t4"
server_key: "603cd454699925701e97e2480ddb247b"
server_key_iv: "ac59d14ca70860172b8ef075ee17b326"
server_key_version: "encryptionkey@0"
I20260620 15:01:48.573275 26530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:48.599439 26530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:48.602134 26530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:48.602327 26530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:48.609181 26530 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:42685
I20260620 15:01:48.609205 26597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:42685 every 8 connection(s)
I20260620 15:01:48.610159 26530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:48.612785 26598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:48.616314 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 26530
I20260620 15:01:48.616463 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:48.616725 7354 external_mini_cluster.cc:1428] Setting key 4a16fe7e43b30f5a34bdc86227f10e51
I20260620 15:01:48.618643 26598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b: Bootstrap starting.
I20260620 15:01:48.621290 26598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:48.622118 26598 log.cc:824] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:48.624176 26598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b: No bootstrap required, opened a new log
Jun 20 15:01:48 dist-test-slave-30t4 krb5kdc[26517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:48.628017 26598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b7079765c4ab1a2e347ebe0a9b57b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42685 } }
I20260620 15:01:48.628290 26598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:48.628350 26598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342b7079765c4ab1a2e347ebe0a9b57b, State: Initialized, Role: FOLLOWER
I20260620 15:01:48.628813 26598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b7079765c4ab1a2e347ebe0a9b57b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42685 } }
I20260620 15:01:48.628971 26598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:48.629055 26598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:48.629154 26598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:48.630223 26598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b7079765c4ab1a2e347ebe0a9b57b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42685 } }
I20260620 15:01:48.630566 26598 leader_election.cc:302] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342b7079765c4ab1a2e347ebe0a9b57b; no voters:
I20260620 15:01:48.630872 26598 leader_election.cc:288] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:48.631074 26603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:48.631280 26603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [term 1 LEADER]: Becoming Leader. State: Replica: 342b7079765c4ab1a2e347ebe0a9b57b, State: Running, Role: LEADER
I20260620 15:01:48.631572 26603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b7079765c4ab1a2e347ebe0a9b57b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42685 } }
I20260620 15:01:48.631940 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:48.618057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.7.46.190:42685)
0620 15:01:48.618461 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:48.618489 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:48.618529 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:01:48.619304 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:48.619325 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:48.619748 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:48.620137 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:48.620753 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:48.622247 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:48.623040 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:48.623709 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:48.623895 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:48.626352 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:48.626358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:48.626373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:48.626405 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:48.628891 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:48.629402 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:48.629408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:48.629415 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:48.629583 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:48.629833 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:48.629836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:48.629838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:48.630346 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:48.630525 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:48.630729 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 15:01:48.631017 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260620 15:01:48.632130 26598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:48.633103 26605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342b7079765c4ab1a2e347ebe0a9b57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b7079765c4ab1a2e347ebe0a9b57b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42685 } } }
I20260620 15:01:48.633220 26605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:48.633569 26604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342b7079765c4ab1a2e347ebe0a9b57b. Latest consensus state: current_term: 1 leader_uuid: "342b7079765c4ab1a2e347ebe0a9b57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b7079765c4ab1a2e347ebe0a9b57b" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 42685 } } }
I20260620 15:01:48.633661 26604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:48.633648 26612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:48.636955 26612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:48.642115 26612 catalog_manager.cc:1347] Generated new cluster ID: 0c1766fd19694e84aef69ea01c7a8bb3
I20260620 15:01:48.642202 26612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:48.651039 26612 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:48.652204 26612 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:48.659262 26612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b: Generated new TSK 0
I20260620 15:01:48.659855 26612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.46.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
Entry for principal kudu/127.7.46.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab.
I20260620 15:01:48.690307 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42685
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:48.800366 26624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:48.800632 26624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:48.800699 26624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:48.804693 26624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:48.804767 26624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:48.804879 26624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:01:48.810588 26624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.129.keytab
--principal=kudu/127.7.46.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:48.811568 26624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:48.812605 26624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:48.821913 26632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:48.821914 26629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:48.821909 26630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:48.822319 26624 server_base.cc:1034] running on GCE node
I20260620 15:01:48.822639 26624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:48.823232 26624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:48.824384 26624 hybrid_clock.cc:648] HybridClock initialized: now 1781967708824369 us; error 25 us; skew 500 ppm
Jun 20 15:01:48 dist-test-slave-30t4 krb5kdc[26517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967708, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:48.827044 26624 init.cc:377] Logged in from keytab as kudu/127.7.46.129@KRBTEST.COM (short username kudu)
I20260620 15:01:48.827898 26624 webserver.cc:460] Webserver started at http://127.7.46.129:33257/ using document root <none> and password file <none>
I20260620 15:01:48.828433 26624 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:48.828497 26624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:48.828771 26624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:48.830674 26624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "101c949ef7d14461b0330118ba2a7083"
format_stamp: "Formatted at 2026-06-20 15:01:48 on dist-test-slave-30t4"
server_key: "7b62c40884b5bfa1a41558615a3ca515"
server_key_iv: "b3f73a75c79c31d03e3728a30bb58daf"
server_key_version: "encryptionkey@0"
I20260620 15:01:48.831230 26624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "101c949ef7d14461b0330118ba2a7083"
format_stamp: "Formatted at 2026-06-20 15:01:48 on dist-test-slave-30t4"
server_key: "7b62c40884b5bfa1a41558615a3ca515"
server_key_iv: "b3f73a75c79c31d03e3728a30bb58daf"
server_key_version: "encryptionkey@0"
I20260620 15:01:48.835140 26624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 15:01:48.837431 26639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:48.838555 26624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 15:01:48.838673 26624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "101c949ef7d14461b0330118ba2a7083"
format_stamp: "Formatted at 2026-06-20 15:01:48 on dist-test-slave-30t4"
server_key: "7b62c40884b5bfa1a41558615a3ca515"
server_key_iv: "b3f73a75c79c31d03e3728a30bb58daf"
server_key_version: "encryptionkey@0"
I20260620 15:01:48.838766 26624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:48.849781 26624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:48.852324 26624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:48.852512 26624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:48.853034 26624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:48.853772 26624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:48.853816 26624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:48.853875 26624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:48.853905 26624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:48.862515 26624 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:43925
I20260620 15:01:48.862536 26752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:43925 every 8 connection(s)
I20260620 15:01:48.863296 26624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:48.865878 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 26624
I20260620 15:01:48.866058 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:48.866370 7354 external_mini_cluster.cc:1428] Setting key 5148ee22ae9f958b8e3f724b70168f3f
Jun 20 15:01:48 dist-test-slave-30t4 krb5kdc[26517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967708, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.129@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.130@KRBTEST.COM; defaulting to no policy
I20260620 15:01:48.876327 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:48.864900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:51027 (local address 127.7.46.190:42685)
0620 15:01:48.865037 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:48.865042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:48.865842 (+ 800us) server_negotiation.cc:407] Connection header received
0620 15:01:48.866888 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:48.866892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:48.866946 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:48.867053 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:48.868155 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:48.868685 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:48.869796 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:48.870035 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:48.870144 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:48.872771 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:48.872775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:48.872777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:48.872798 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:48.874218 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:48.874839 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:48.874845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:48.874850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:48.874944 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:48.875260 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:48.875262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:48.875264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:48.875426 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:48.875511 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:48.876112 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 15:01:48.876210 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.130@KRBTEST.COM" created.
I20260620 15:01:48.877290 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:48.865302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:42685 (local address 127.7.46.129:51027)
0620 15:01:48.865704 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:48.865742 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:48.866692 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:48.867216 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:48.867223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:48.867550 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:48.868011 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:48.868024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:48.868812 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:48.868816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:48.869668 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:48.869677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:48.870181 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:48.870185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:48.870348 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:48.870910 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:48.870926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:48.872607 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:48.874347 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:48.874352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:48.874365 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:48.874717 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:48.875044 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:48.875046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:48.875048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:48.875171 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:48.875528 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:48.875532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:48.875817 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 15:01:48.876070 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 15:01:48.876326 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20260620 15:01:48.878495 26753 heartbeater.cc:347] Connected to a master server at 127.7.46.190:42685
I20260620 15:01:48.878747 26753 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:48.879314 26753 heartbeater.cc:511] Master 127.7.46.190:42685 requested a full tablet report, sending...
I20260620 15:01:48.880857 26562 ts_manager.cc:194] Registered new tserver with Master: 101c949ef7d14461b0330118ba2a7083 (127.7.46.129:43925)
I20260620 15:01:48.881968 26562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.129@KRBTEST.COM'} at 127.7.46.129:51027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
Entry for principal kudu/127.7.46.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab.
I20260620 15:01:48.892925 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42685
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:48.999807 26758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:49.000090 26758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:49.000162 26758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:49.004109 26758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:49.004191 26758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:49.004281 26758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:01:49.009263 26758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.130.keytab
--principal=kudu/127.7.46.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:49.010231 26758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:49.011195 26758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:49.020623 26763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:49.020694 26758 server_base.cc:1034] running on GCE node
W20260620 15:01:49.020833 26764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:49.021028 26766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:49.021441 26758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:49.022020 26758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:49.023176 26758 hybrid_clock.cc:648] HybridClock initialized: now 1781967709023155 us; error 30 us; skew 500 ppm
Jun 20 15:01:49 dist-test-slave-30t4 krb5kdc[26517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:49.025734 26758 init.cc:377] Logged in from keytab as kudu/127.7.46.130@KRBTEST.COM (short username kudu)
I20260620 15:01:49.026451 26758 webserver.cc:460] Webserver started at http://127.7.46.130:43595/ using document root <none> and password file <none>
I20260620 15:01:49.026924 26758 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:49.026984 26758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:49.027132 26758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:49.029109 26758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "53111ca5e695483a874669c19cb0a91a"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "9877b467fe36d486cbd6fa131b9b242d"
server_key_iv: "7edb31a88bd7df4cee98825520654e19"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.029587 26758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "53111ca5e695483a874669c19cb0a91a"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "9877b467fe36d486cbd6fa131b9b242d"
server_key_iv: "7edb31a88bd7df4cee98825520654e19"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.033136 26758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:49.035367 26773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.036553 26758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:49.036664 26758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "53111ca5e695483a874669c19cb0a91a"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "9877b467fe36d486cbd6fa131b9b242d"
server_key_iv: "7edb31a88bd7df4cee98825520654e19"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.036760 26758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:49.049072 26758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:49.052053 26758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:49.052250 26758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:49.052776 26758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:49.053520 26758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:49.053565 26758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.053629 26758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:49.053669 26758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.061703 26758 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:36147
I20260620 15:01:49.061725 26886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:36147 every 8 connection(s)
I20260620 15:01:49.062491 26758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:49.068296 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 26758
I20260620 15:01:49.068432 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:49.068732 7354 external_mini_cluster.cc:1428] Setting key b25d9e4dd41cfeace1fcd03931b10e07
Jun 20 15:01:49 dist-test-slave-30t4 krb5kdc[26517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.130@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:49.076117 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.064200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:35979 (local address 127.7.46.190:42685)
0620 15:01:49.064333 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.064337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.065073 (+ 736us) server_negotiation.cc:407] Connection header received
0620 15:01:49.066294 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.066298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.066351 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.066459 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:49.067492 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.068129 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.069466 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.069681 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.069760 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.072398 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:49.072401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:49.072403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:49.072424 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:49.074105 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:49.074617 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:49.074625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:49.074628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:49.074714 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:49.075047 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:49.075051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:49.075052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:49.075233 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:49.075340 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.075832 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.075947 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:01:49.076884 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.064545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:42685 (local address 127.7.46.130:35979)
0620 15:01:49.064942 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.064982 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:49.066099 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:49.066628 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:49.066634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:49.066939 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:49.067349 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.067361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.068335 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.068338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.069325 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.069335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.069832 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.069839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.069986 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.070535 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:49.070551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:49.072239 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:49.074230 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:49.074239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:49.074255 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:49.074514 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:49.074807 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:49.074809 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:49.074811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:49.074944 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:49.075338 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:49.075345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:49.075565 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 15:01:49.075744 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 15:01:49.076053 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260620 15:01:49.078001 26887 heartbeater.cc:347] Connected to a master server at 127.7.46.190:42685
I20260620 15:01:49.078235 26887 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.46.131@KRBTEST.COM; defaulting to no policy
I20260620 15:01:49.078713 26887 heartbeater.cc:511] Master 127.7.46.190:42685 requested a full tablet report, sending...
I20260620 15:01:49.079665 26562 ts_manager.cc:194] Registered new tserver with Master: 53111ca5e695483a874669c19cb0a91a (127.7.46.130:36147)
I20260620 15:01:49.080221 26562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.130@KRBTEST.COM'} at 127.7.46.130:35979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
Entry for principal kudu/127.7.46.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab.
I20260620 15:01:49.094807 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42685
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:49.201541 26892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:49.201795 26892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:49.201866 26892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:49.205829 26892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:49.205905 26892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:49.206003 26892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:01:49.211148 26892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.131.keytab
--principal=kudu/127.7.46.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:42685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:49.212162 26892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:49.213110 26892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:49.222155 26897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:49.222211 26892 server_base.cc:1034] running on GCE node
W20260620 15:01:49.222193 26898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:49.222354 26900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:49.222707 26892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:49.223253 26892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:49.224423 26892 hybrid_clock.cc:648] HybridClock initialized: now 1781967709224401 us; error 33 us; skew 500 ppm
Jun 20 15:01:49 dist-test-slave-30t4 krb5kdc[26517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:49.226882 26892 init.cc:377] Logged in from keytab as kudu/127.7.46.131@KRBTEST.COM (short username kudu)
I20260620 15:01:49.227612 26892 webserver.cc:460] Webserver started at http://127.7.46.131:38363/ using document root <none> and password file <none>
I20260620 15:01:49.228175 26892 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:49.228236 26892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:49.228403 26892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:49.230199 26892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "5d9c1c1520344342926982698ab33bad"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "cfb95529a944936e70be650e990fa8de"
server_key_iv: "ba335a06435cb081f19b6ce42f456e15"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.230693 26892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "5d9c1c1520344342926982698ab33bad"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "cfb95529a944936e70be650e990fa8de"
server_key_iv: "ba335a06435cb081f19b6ce42f456e15"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.234376 26892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 15:01:49.236609 26907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.237704 26892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:49.237799 26892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "5d9c1c1520344342926982698ab33bad"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "cfb95529a944936e70be650e990fa8de"
server_key_iv: "ba335a06435cb081f19b6ce42f456e15"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.237875 26892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:49.262267 26892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:49.265512 26892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:49.265659 26892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:49.266177 26892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:49.266914 26892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:49.266959 26892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.266999 26892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:49.267014 26892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.275341 26892 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:45043
I20260620 15:01:49.275384 27020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:45043 every 8 connection(s)
I20260620 15:01:49.276315 26892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:49.280514 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 26892
I20260620 15:01:49.280654 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:49.280972 7354 external_mini_cluster.cc:1428] Setting key e5937f03836eb9445a944f24b32582f4
Jun 20 15:01:49 dist-test-slave-30t4 krb5kdc[26517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.131@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:01:49.289471 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.278071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:43479 (local address 127.7.46.190:42685)
0620 15:01:49.278253 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.278258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.279066 (+ 808us) server_negotiation.cc:407] Connection header received
0620 15:01:49.280353 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.280356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.280422 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.280499 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:49.281810 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.282543 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.283840 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.284055 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.284135 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.286317 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:49.286321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:49.286323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:49.286346 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:49.287889 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:49.288293 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:49.288295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:49.288297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:49.288367 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:49.288642 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:49.288644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:49.288645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:49.288799 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:49.288873 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.289263 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.289355 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:01:49.290166 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.278503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:42685 (local address 127.7.46.131:43479)
0620 15:01:49.278910 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.278964 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:49.280131 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:49.280705 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:49.280715 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:49.281185 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:49.281701 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.281714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.282662 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.282666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.283683 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.283692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.284175 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.284178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.284312 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.284807 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:49.284822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:49.286153 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:49.287979 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:49.287983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:49.287993 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:49.288214 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:49.288444 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:49.288446 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:49.288448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:49.288567 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:49.288882 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:49.288887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:49.289077 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 15:01:49.289216 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:01:49.289426 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20260620 15:01:49.291213 27021 heartbeater.cc:347] Connected to a master server at 127.7.46.190:42685
I20260620 15:01:49.291389 27021 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:49.291823 27021 heartbeater.cc:511] Master 127.7.46.190:42685 requested a full tablet report, sending...
I20260620 15:01:49.292740 26562 ts_manager.cc:194] Registered new tserver with Master: 5d9c1c1520344342926982698ab33bad (127.7.46.131:45043)
I20260620 15:01:49.293244 26562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.46.131@KRBTEST.COM'} at 127.7.46.131:43479
I20260620 15:01:49.294575 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:01:49.303349 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.296416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.7.46.190:42685)
0620 15:01:49.296555 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.296560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.296712 (+ 152us) server_negotiation.cc:407] Connection header received
0620 15:01:49.296848 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.296851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.296889 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.296997 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:49.297620 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.298108 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.298925 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.299155 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.299259 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.300526 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:49.300529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:49.300532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:49.300555 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:49.301775 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:49.302169 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:49.302174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:49.302175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:49.302255 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:49.302520 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:49.302524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:49.302526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:49.302724 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:49.302859 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.303086 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.303197 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:01:49.305871 26562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:01:49.308773 26562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:01:49.322219 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.317264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:36147 (local address 127.0.0.1:36200)
0620 15:01:49.317595 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.317608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:49.317692 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:49.318392 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:49.318395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:49.318415 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:49.318511 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.318517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.319493 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.319498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.320827 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.320837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.321715 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.321718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.321817 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.321852 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:01:49.321963 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:01:49.322076 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":110,"threads_started":1}
I20260620 15:01:49.322232 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.317420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.7.46.130:36147)
0620 15:01:49.317935 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.317946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.317968 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:01:49.318040 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.318045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.318238 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.318421 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:49.318633 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.319353 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.320981 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.321591 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.321715 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.321862 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.321999 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.322089 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":91,"threads_started":1}
I20260620 15:01:49.322435 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.316624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:43925 (local address 127.0.0.1:44684)
0620 15:01:49.317227 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.317243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:49.317329 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:49.318163 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:49.318169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:49.318193 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:49.318316 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.318322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.319899 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.319903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.321038 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.321048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.322049 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.322052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.322144 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.322166 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:01:49.322217 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 15:01:49.322286 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":106,"threads_started":1}
I20260620 15:01:49.322580 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.316547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:45043 (local address 127.0.0.1:35346)
0620 15:01:49.316872 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.316924 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:49.317036 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:49.318314 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:49.318317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:49.318331 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:49.318423 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.318428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.319937 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.319940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.320972 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.320981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.322216 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.322219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.322307 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.322325 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 15:01:49.322386 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 15:01:49.322440 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":101,"threads_started":1}
I20260620 15:01:49.322947 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.316874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.7.46.131:45043)
0620 15:01:49.317686 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.317697 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.317720 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:01:49.317795 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.317801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.317999 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.318143 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:49.318582 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.319739 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.321093 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.321899 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.322026 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.322158 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.322721 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.322811 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":100,"threads_started":1}
I20260620 15:01:49.323706 27036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.317317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.7.46.129:43925)
0620 15:01:49.317721 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.317731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.317750 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:01:49.317808 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.317812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.317991 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.318477 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:49.318627 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.319746 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.321145 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.321899 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.322694 (+ 795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.322827 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.322870 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.322948 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":127,"threads_started":1}
I20260620 15:01:49.325623 26955 tablet_service.cc:1467] Processing CreateTablet for tablet e788c350f37f405bac7ce3428899d78f (DEFAULT_TABLE table=test-table [id=ebcfaaf45cba4834a265067f9cfe0588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:49.325623 26821 tablet_service.cc:1467] Processing CreateTablet for tablet e788c350f37f405bac7ce3428899d78f (DEFAULT_TABLE table=test-table [id=ebcfaaf45cba4834a265067f9cfe0588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:49.325873 26687 tablet_service.cc:1467] Processing CreateTablet for tablet e788c350f37f405bac7ce3428899d78f (DEFAULT_TABLE table=test-table [id=ebcfaaf45cba4834a265067f9cfe0588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:49.326711 26955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e788c350f37f405bac7ce3428899d78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:49.326704 26821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e788c350f37f405bac7ce3428899d78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:49.326714 26687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e788c350f37f405bac7ce3428899d78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:49.332937 27038 tablet_bootstrap.cc:492] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083: Bootstrap starting.
I20260620 15:01:49.334162 27040 tablet_bootstrap.cc:492] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: Bootstrap starting.
I20260620 15:01:49.335433 27038 tablet_bootstrap.cc:654] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:49.335898 27040 tablet_bootstrap.cc:654] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:49.336097 27039 tablet_bootstrap.cc:492] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a: Bootstrap starting.
I20260620 15:01:49.336282 27038 log.cc:824] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:49.337065 27040 log.cc:824] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:49.338218 27038 tablet_bootstrap.cc:492] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083: No bootstrap required, opened a new log
I20260620 15:01:49.338423 27038 ts_tablet_manager.cc:1397] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 15:01:49.338603 27039 tablet_bootstrap.cc:654] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:49.339162 27040 tablet_bootstrap.cc:492] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: No bootstrap required, opened a new log
I20260620 15:01:49.339342 27040 ts_tablet_manager.cc:1397] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 15:01:49.339354 27039 log.cc:824] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:49.341192 27039 tablet_bootstrap.cc:492] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a: No bootstrap required, opened a new log
I20260620 15:01:49.341356 27039 ts_tablet_manager.cc:1397] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 15:01:49.341924 27038 raft_consensus.cc:348] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.342262 27038 raft_consensus.cc:374] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:49.342319 27038 raft_consensus.cc:729] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101c949ef7d14461b0330118ba2a7083, State: Initialized, Role: FOLLOWER
I20260620 15:01:49.342730 27038 consensus_queue.cc:260] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.343017 27040 raft_consensus.cc:348] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.343271 27040 raft_consensus.cc:374] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:49.343335 27040 raft_consensus.cc:729] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9c1c1520344342926982698ab33bad, State: Initialized, Role: FOLLOWER
I20260620 15:01:49.343394 26753 heartbeater.cc:503] Master 127.7.46.190:42685 was elected leader, sending a full tablet report...
I20260620 15:01:49.343644 27038 ts_tablet_manager.cc:1428] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 15:01:49.343712 27040 consensus_queue.cc:260] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.344662 27040 ts_tablet_manager.cc:1428] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 15:01:49.345510 27039 raft_consensus.cc:348] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.345726 27039 raft_consensus.cc:374] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:49.345781 27039 raft_consensus.cc:729] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53111ca5e695483a874669c19cb0a91a, State: Initialized, Role: FOLLOWER
I20260620 15:01:49.346241 27021 heartbeater.cc:503] Master 127.7.46.190:42685 was elected leader, sending a full tablet report...
I20260620 15:01:49.346189 27039 consensus_queue.cc:260] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.346781 26887 heartbeater.cc:503] Master 127.7.46.190:42685 was elected leader, sending a full tablet report...
I20260620 15:01:49.347084 27039 ts_tablet_manager.cc:1428] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 15:01:49.364368 26754 tablet.cc:2369] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:49.399011 27045 raft_consensus.cc:482] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:49.399181 27045 raft_consensus.cc:504] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.400285 27045 leader_election.cc:288] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 101c949ef7d14461b0330118ba2a7083 (127.7.46.129:43925), 53111ca5e695483a874669c19cb0a91a (127.7.46.130:36147)
I20260620 15:01:49.405349 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.400674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:50199 (local address 127.7.46.130:36147)
0620 15:01:49.400799 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.400804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.400921 (+ 117us) server_negotiation.cc:407] Connection header received
0620 15:01:49.400994 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.400998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.401048 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.401117 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:49.401413 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.402190 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.404114 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.404710 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.404831 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.404881 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.405157 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.405222 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:01:49.405349 27036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.400520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:55453 (local address 127.7.46.129:43925)
0620 15:01:49.400634 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.400639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.400778 (+ 139us) server_negotiation.cc:407] Connection header received
0620 15:01:49.400845 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.400854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.400927 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.401038 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:49.401413 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.402192 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.404114 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.404710 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.404831 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.404881 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.405157 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.405222 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 15:01:49.405382 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.400461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:43925 (local address 127.7.46.131:55453)
0620 15:01:49.400632 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.400647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:49.400738 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:49.401074 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:49.401077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:49.401124 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:49.401266 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.401274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.402324 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.402328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.403990 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.404001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.404837 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.404841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.405007 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.405053 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:01:49.405138 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:01:49.405230 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":213,"spinlock_wait_cycles":11904}
I20260620 15:01:49.405397 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.400502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:36147 (local address 127.7.46.131:50199)
0620 15:01:49.400829 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.400841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:49.400924 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:49.401152 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:49.401154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:49.401184 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:49.401282 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.401287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.402324 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.402327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.404002 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.404008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.404837 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.404841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.405010 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.405050 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:01:49.405128 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 15:01:49.405230 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":275,"spinlock_wait_cycles":896,"thread_start_us":128,"threads_started":1}
I20260620 15:01:49.406030 26707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e788c350f37f405bac7ce3428899d78f" candidate_uuid: "5d9c1c1520344342926982698ab33bad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101c949ef7d14461b0330118ba2a7083" is_pre_election: true
I20260620 15:01:49.406297 26707 raft_consensus.cc:2454] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9c1c1520344342926982698ab33bad in term 0.
I20260620 15:01:49.406301 26841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e788c350f37f405bac7ce3428899d78f" candidate_uuid: "5d9c1c1520344342926982698ab33bad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53111ca5e695483a874669c19cb0a91a" is_pre_election: true
I20260620 15:01:49.406559 26841 raft_consensus.cc:2454] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9c1c1520344342926982698ab33bad in term 0.
I20260620 15:01:49.406761 26908 leader_election.cc:302] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101c949ef7d14461b0330118ba2a7083, 5d9c1c1520344342926982698ab33bad; no voters:
I20260620 15:01:49.407033 27045 raft_consensus.cc:2781] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:01:49.407107 27045 raft_consensus.cc:482] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:01:49.407137 27045 raft_consensus.cc:3037] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:49.408164 27045 raft_consensus.cc:504] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.408493 27045 leader_election.cc:288] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [CANDIDATE]: Term 1 election: Requested vote from peers 101c949ef7d14461b0330118ba2a7083 (127.7.46.129:43925), 53111ca5e695483a874669c19cb0a91a (127.7.46.130:36147)
I20260620 15:01:49.408842 26707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e788c350f37f405bac7ce3428899d78f" candidate_uuid: "5d9c1c1520344342926982698ab33bad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101c949ef7d14461b0330118ba2a7083"
I20260620 15:01:49.408967 26707 raft_consensus.cc:3037] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:49.409922 26707 raft_consensus.cc:2454] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9c1c1520344342926982698ab33bad in term 1.
I20260620 15:01:49.410174 26841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e788c350f37f405bac7ce3428899d78f" candidate_uuid: "5d9c1c1520344342926982698ab33bad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53111ca5e695483a874669c19cb0a91a"
I20260620 15:01:49.410236 26908 leader_election.cc:302] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101c949ef7d14461b0330118ba2a7083, 5d9c1c1520344342926982698ab33bad; no voters:
I20260620 15:01:49.410298 26841 raft_consensus.cc:3037] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:49.411327 26841 raft_consensus.cc:2454] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9c1c1520344342926982698ab33bad in term 1.
I20260620 15:01:49.411540 27045 raft_consensus.cc:2781] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:49.411806 27045 raft_consensus.cc:686] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 1 LEADER]: Becoming Leader. State: Replica: 5d9c1c1520344342926982698ab33bad, State: Running, Role: LEADER
I20260620 15:01:49.412110 27045 consensus_queue.cc:237] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } }
I20260620 15:01:49.415066 26561 catalog_manager.cc:5495] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d9c1c1520344342926982698ab33bad (127.7.46.131). New cstate: current_term: 1 leader_uuid: "5d9c1c1520344342926982698ab33bad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9c1c1520344342926982698ab33bad" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 45043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 } health_report { overall_health: UNKNOWN } } }
I20260620 15:01:49.443527 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.439466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.7.46.131:45043)
0620 15:01:49.439628 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.439633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.439669 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:01:49.439748 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.439750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.439815 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.439922 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:49.440190 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.440975 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.442047 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.442287 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.442392 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.442743 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:49.443135 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:01:49.443260 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.443322 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.443373 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:01:49.448798 26707 raft_consensus.cc:1261] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083 [term 1 FOLLOWER]: Refusing update from remote peer 5d9c1c1520344342926982698ab33bad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:49.448798 26841 raft_consensus.cc:1261] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 1 FOLLOWER]: Refusing update from remote peer 5d9c1c1520344342926982698ab33bad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:49.449440 27045 consensus_queue.cc:1035] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [LEADER]: Connected to new peer: Peer: permanent_uuid: "101c949ef7d14461b0330118ba2a7083" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:49.449642 27048 consensus_queue.cc:1035] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [LEADER]: Connected to new peer: Peer: permanent_uuid: "53111ca5e695483a874669c19cb0a91a" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 36147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:49.464557 26562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:49.464720 26562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:49.466996 26562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 342b7079765c4ab1a2e347ebe0a9b57b: Sending DeleteTablet for 3 replicas of tablet e788c350f37f405bac7ce3428899d78f
I20260620 15:01:49.467721 26955 tablet_service.cc:1514] Processing DeleteTablet for tablet e788c350f37f405bac7ce3428899d78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:49 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:35346
I20260620 15:01:49.467716 26687 tablet_service.cc:1514] Processing DeleteTablet for tablet e788c350f37f405bac7ce3428899d78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:49 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:44684
I20260620 15:01:49.468202 27062 tablet_replica.cc:330] T e788c350f37f405bac7ce3428899d78f P 101c949ef7d14461b0330118ba2a7083: stopping tablet replica
I20260620 15:01:49.468307 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 26624
I20260620 15:01:49.469990 27061 tablet_replica.cc:330] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: stopping tablet replica
I20260620 15:01:49.470248 27061 raft_consensus.cc:2229] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:01:49.470604 27061 raft_consensus.cc:2258] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:49.470582 26821 tablet_service.cc:1514] Processing DeleteTablet for tablet e788c350f37f405bac7ce3428899d78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:49 UTC) from {username='kudu', principal='kudu/127.7.46.190@KRBTEST.COM'} at 127.0.0.1:36200
I20260620 15:01:49.471752 27061 ts_tablet_manager.cc:1905] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:49.473901 27061 ts_tablet_manager.cc:1918] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:49.474016 27061 log.cc:1196] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/e788c350f37f405bac7ce3428899d78f
I20260620 15:01:49.474249 27063 tablet_replica.cc:330] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a: stopping tablet replica
I20260620 15:01:49.474367 27061 ts_tablet_manager.cc:1939] T e788c350f37f405bac7ce3428899d78f P 5d9c1c1520344342926982698ab33bad: Deleting consensus metadata
I20260620 15:01:49.474655 27063 raft_consensus.cc:2229] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:49.474974 27063 raft_consensus.cc:2258] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:49.475440 26547 catalog_manager.cc:4841] TS 5d9c1c1520344342926982698ab33bad (127.7.46.131:45043): tablet e788c350f37f405bac7ce3428899d78f (table test-table [id=ebcfaaf45cba4834a265067f9cfe0588]) successfully deleted
W20260620 15:01:49.476186 26546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:43925 (error 108)
I20260620 15:01:49.476548 27063 ts_tablet_manager.cc:1905] T e788c350f37f405bac7ce3428899d78f P 53111ca5e695483a874669c19cb0a91a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:49.476940 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 26758
I20260620 15:01:49.478108 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.476936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:43925 (local address 127.0.0.1:44700)
0620 15:01:49.477948 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.478030 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":931}
W20260620 15:01:49.478299 26546 catalog_manager.cc:4584] TS 101c949ef7d14461b0330118ba2a7083 (127.7.46.129:43925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e788c350f37f405bac7ce3428899d78f: Network error: Client connection negotiation failed: client connection to 127.7.46.129:43925: connect: Connection refused (error 111)
I20260620 15:01:49.482676 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 26892
I20260620 15:01:49.482839 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.482588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:36147 (local address 127.0.0.1:36202)
0620 15:01:49.482729 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.482774 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.130:36147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 15:01:49.487478 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 26530
2026-06-20T15:01:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2202 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T15:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:49Z Disabled control of system clock
I20260620 15:01:49.516778 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43537
--webserver_interface=127.7.46.190
--webserver_port=0
--builtin_ntp_servers=127.7.46.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:43537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:01:49.622457 27068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:49.622695 27068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:49.622730 27068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:49.626559 27068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:49.626605 27068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:49.626619 27068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:49.626636 27068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:49.626650 27068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:49.631155 27068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:43537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:49.632053 27068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:49.633279 27068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:49.641074 27073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:49.641100 27074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:49.641207 27068 server_base.cc:1034] running on GCE node
W20260620 15:01:49.641165 27076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:49.641911 27068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:49.642890 27068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:49.644075 27068 hybrid_clock.cc:648] HybridClock initialized: now 1781967709644044 us; error 44 us; skew 500 ppm
I20260620 15:01:49.645784 27068 webserver.cc:460] Webserver started at http://127.7.46.190:37065/ using document root <none> and password file <none>
I20260620 15:01:49.646328 27068 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:49.646415 27068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:49.646632 27068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:49.648586 27068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "63cb90db25bc4ad294ddccd392f9ffef"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "39881328a20f981fb8f593ea6009ff60"
server_key_iv: "bc603b54870297b23c670702cd53dd8b"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.649120 27068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "63cb90db25bc4ad294ddccd392f9ffef"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "39881328a20f981fb8f593ea6009ff60"
server_key_iv: "bc603b54870297b23c670702cd53dd8b"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.652581 27068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:49.654733 27082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.655902 27068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:49.656031 27068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "63cb90db25bc4ad294ddccd392f9ffef"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "39881328a20f981fb8f593ea6009ff60"
server_key_iv: "bc603b54870297b23c670702cd53dd8b"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.656132 27068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:49.674626 27068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:49.675263 27068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:49.675457 27068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:49.682080 27068 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:43537
I20260620 15:01:49.682085 27134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:43537 every 8 connection(s)
I20260620 15:01:49.682941 27068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:49.685844 27135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:49.690902 27135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef: Bootstrap starting.
I20260620 15:01:49.692019 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 27068
I20260620 15:01:49.692139 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:49.692446 7354 external_mini_cluster.cc:1428] Setting key 13a239028825b23592dfb9c04a23d54a
I20260620 15:01:49.693578 27135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:49.694290 27135 log.cc:824] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:49.696364 27135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef: No bootstrap required, opened a new log
I20260620 15:01:49.700551 27135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cb90db25bc4ad294ddccd392f9ffef" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43537 } }
I20260620 15:01:49.700791 27135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:49.700867 27135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cb90db25bc4ad294ddccd392f9ffef, State: Initialized, Role: FOLLOWER
I20260620 15:01:49.701269 27135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cb90db25bc4ad294ddccd392f9ffef" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43537 } }
I20260620 15:01:49.701417 27135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:49.701511 27135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:49.701614 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.693570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.7.46.190:43537)
0620 15:01:49.694199 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.694216 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.694248 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:01:49.695048 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.695077 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.695412 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.695674 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:49.696054 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.697067 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.698266 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.698708 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.698820 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.699660 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:49.699664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:49.699676 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:49.700109 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:49.700164 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:49.700185 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:49.700259 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.700419 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.700698 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":125,"threads_started":1}
I20260620 15:01:49.701638 27135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:49.702721 27135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cb90db25bc4ad294ddccd392f9ffef" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43537 } }
I20260620 15:01:49.703042 27135 leader_election.cc:302] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63cb90db25bc4ad294ddccd392f9ffef; no voters:
I20260620 15:01:49.703325 27135 leader_election.cc:288] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:49.703420 27140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:49.703648 27140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [term 1 LEADER]: Becoming Leader. State: Replica: 63cb90db25bc4ad294ddccd392f9ffef, State: Running, Role: LEADER
I20260620 15:01:49.704193 27140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cb90db25bc4ad294ddccd392f9ffef" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43537 } }
I20260620 15:01:49.704536 27135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:49.705674 27142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63cb90db25bc4ad294ddccd392f9ffef. Latest consensus state: current_term: 1 leader_uuid: "63cb90db25bc4ad294ddccd392f9ffef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cb90db25bc4ad294ddccd392f9ffef" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43537 } } }
I20260620 15:01:49.705802 27142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:49.706225 27149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:49.706210 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63cb90db25bc4ad294ddccd392f9ffef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cb90db25bc4ad294ddccd392f9ffef" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 43537 } } }
I20260620 15:01:49.706820 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:49.709458 27149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:49.714453 27149 catalog_manager.cc:1347] Generated new cluster ID: e5378809e1a84448b39c1abaf2e5dcc1
I20260620 15:01:49.714521 27149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:49.729094 27149 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:49.730275 27149 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:49.745770 27149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef: Generated new TSK 0
I20260620 15:01:49.746477 27149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:49.754830 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.129:0
--local_ip_for_outbound_sockets=127.7.46.129
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43537
--builtin_ntp_servers=127.7.46.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:01:49.863044 27159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:49.863277 27159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:49.863348 27159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:49.867316 27159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:49.867365 27159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:49.867453 27159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.129
I20260620 15:01:49.872066 27159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.46.129
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.129
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:49.873029 27159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:49.874307 27159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:01:49.883659 27159 server_base.cc:1034] running on GCE node
W20260620 15:01:49.883689 27167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:49.883709 27164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:49.883689 27165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:49.884325 27159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:49.884922 27159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:49.886106 27159 hybrid_clock.cc:648] HybridClock initialized: now 1781967709886074 us; error 44 us; skew 500 ppm
I20260620 15:01:49.887859 27159 webserver.cc:460] Webserver started at http://127.7.46.129:34675/ using document root <none> and password file <none>
I20260620 15:01:49.888394 27159 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:49.888492 27159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:49.888717 27159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:49.890640 27159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/data/instance:
uuid: "2c01162261504690ad168fe64e52df78"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "782024ac8f78bf5cfb0e6f9ffed3f83b"
server_key_iv: "48bdc187de951db96e37c6946e263c11"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.891177 27159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance:
uuid: "2c01162261504690ad168fe64e52df78"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "782024ac8f78bf5cfb0e6f9ffed3f83b"
server_key_iv: "48bdc187de951db96e37c6946e263c11"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.894641 27159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 15:01:49.896893 27173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.898001 27159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:01:49.898146 27159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/wal
uuid: "2c01162261504690ad168fe64e52df78"
format_stamp: "Formatted at 2026-06-20 15:01:49 on dist-test-slave-30t4"
server_key: "782024ac8f78bf5cfb0e6f9ffed3f83b"
server_key_iv: "48bdc187de951db96e37c6946e263c11"
server_key_version: "encryptionkey@0"
I20260620 15:01:49.898253 27159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:49.909260 27159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:49.909814 27159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:49.909994 27159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:49.910524 27159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:49.911361 27159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:49.911432 27159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.911505 27159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:49.911557 27159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:49.919827 27159 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.129:45959
I20260620 15:01:49.919857 27286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.129:45959 every 8 connection(s)
I20260620 15:01:49.920663 27159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:49.929121 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.922314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.129:59511 (local address 127.7.46.190:43537)
0620 15:01:49.922460 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:49.922465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:49.923443 (+ 978us) server_negotiation.cc:407] Connection header received
0620 15:01:49.924707 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:49.924709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:49.924755 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:49.924828 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:49.925600 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.926123 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.927200 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.927434 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.927548 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.928614 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:49.928618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:49.928621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:49.928633 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:49.928660 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:49.928668 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:49.928743 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:49.928903 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 15:01:49.928978 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:01:49.929885 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:49.922727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43537 (local address 127.7.46.129:59511)
0620 15:01:49.923294 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:49.923329 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:49.924524 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:49.924972 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:49.924978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:49.925053 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:49.925483 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.925495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.926224 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.926227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.927070 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:49.927079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:49.927576 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:49.927580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:49.927744 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:49.928341 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:49.928343 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:49.928378 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:49.928457 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:49.928756 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:49.928767 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:49.928774 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:01:49.928897 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 15:01:49.929134 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":102,"threads_started":1}
I20260620 15:01:49.930819 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 27159
I20260620 15:01:49.930940 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:49.931037 27287 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43537
I20260620 15:01:49.931226 7354 external_mini_cluster.cc:1428] Setting key 520a0e86a5529576d12445b5d4f9d211
I20260620 15:01:49.931252 27287 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:49.931804 27287 heartbeater.cc:511] Master 127.7.46.190:43537 requested a full tablet report, sending...
I20260620 15:01:49.933007 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.130:0
--local_ip_for_outbound_sockets=127.7.46.130
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43537
--builtin_ntp_servers=127.7.46.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:01:49.933192 27099 ts_manager.cc:194] Registered new tserver with Master: 2c01162261504690ad168fe64e52df78 (127.7.46.129:45959)
I20260620 15:01:49.934266 27099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.129:59511
W20260620 15:01:50.041281 27290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:50.041527 27290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:50.041563 27290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:50.045449 27290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:50.045492 27290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:50.045543 27290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.130
I20260620 15:01:50.050112 27290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.46.130
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.130
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:50.051005 27290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:50.052261 27290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:50.061147 27298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:50.061146 27295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:50.061147 27296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:50.061408 27290 server_base.cc:1034] running on GCE node
I20260620 15:01:50.061825 27290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:50.062356 27290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:50.063549 27290 hybrid_clock.cc:648] HybridClock initialized: now 1781967710063538 us; error 44 us; skew 500 ppm
I20260620 15:01:50.069886 27290 webserver.cc:460] Webserver started at http://127.7.46.130:35603/ using document root <none> and password file <none>
I20260620 15:01:50.070437 27290 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:50.070534 27290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:50.070755 27290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:50.072748 27290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/data/instance:
uuid: "536ee079a473406593e7e63947481d6b"
format_stamp: "Formatted at 2026-06-20 15:01:50 on dist-test-slave-30t4"
server_key: "ffd0ce93f3672703b1cefa6064ea5826"
server_key_iv: "3c3ec7ba5ea79f3e9598780542444774"
server_key_version: "encryptionkey@0"
I20260620 15:01:50.073278 27290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance:
uuid: "536ee079a473406593e7e63947481d6b"
format_stamp: "Formatted at 2026-06-20 15:01:50 on dist-test-slave-30t4"
server_key: "ffd0ce93f3672703b1cefa6064ea5826"
server_key_iv: "3c3ec7ba5ea79f3e9598780542444774"
server_key_version: "encryptionkey@0"
I20260620 15:01:50.076711 27290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:50.078970 27304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:50.080230 27290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:50.080353 27290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/wal
uuid: "536ee079a473406593e7e63947481d6b"
format_stamp: "Formatted at 2026-06-20 15:01:50 on dist-test-slave-30t4"
server_key: "ffd0ce93f3672703b1cefa6064ea5826"
server_key_iv: "3c3ec7ba5ea79f3e9598780542444774"
server_key_version: "encryptionkey@0"
I20260620 15:01:50.080463 27290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:50.097489 27290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:50.098068 27290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:50.098260 27290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:50.098800 27290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:50.099690 27290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:50.099761 27290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:50.099884 27290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:50.099936 27290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:50.109426 27290 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.130:43055
I20260620 15:01:50.109453 27417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.130:43055 every 8 connection(s)
I20260620 15:01:50.110253 27290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:50.118933 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 27290
I20260620 15:01:50.118956 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.111917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.130:45425 (local address 127.7.46.190:43537)
0620 15:01:50.112048 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.112051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.113019 (+ 968us) server_negotiation.cc:407] Connection header received
0620 15:01:50.114368 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.114371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.114432 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.114511 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:50.115340 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.115951 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.117048 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.117286 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.117395 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.118464 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:50.118468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:50.118471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:50.118482 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:50.118503 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:50.118519 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:50.118593 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.118748 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.118807 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:01:50.119086 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:50.119390 7354 external_mini_cluster.cc:1428] Setting key d5fae4b9d94d0d299be4d04a4ec0720c
I20260620 15:01:50.119887 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.112281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43537 (local address 127.7.46.130:45425)
0620 15:01:50.112881 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:50.112918 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:50.114154 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:50.114682 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:50.114688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:50.114768 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:50.115204 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.115215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.116068 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.116079 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.116921 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.116929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.117429 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.117433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.117600 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.118203 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:50.118205 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:50.118242 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:50.118299 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:50.118606 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:50.118615 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:50.118620 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:01:50.118747 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:01:50.119000 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":118,"threads_started":1}
I20260620 15:01:50.120935 27418 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43537
I20260620 15:01:50.121162 27418 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:50.121471 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.46.131:0
--local_ip_for_outbound_sockets=127.7.46.131
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43537
--builtin_ntp_servers=127.7.46.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:01:50.121726 27418 heartbeater.cc:511] Master 127.7.46.190:43537 requested a full tablet report, sending...
I20260620 15:01:50.122771 27098 ts_manager.cc:194] Registered new tserver with Master: 536ee079a473406593e7e63947481d6b (127.7.46.130:43055)
I20260620 15:01:50.123356 27098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.130:45425
W20260620 15:01:50.243016 27421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:50.243224 27421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:50.243278 27421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:50.247229 27421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:50.247275 27421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:50.247370 27421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.46.131
I20260620 15:01:50.251932 27421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.46.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.46.131
--webserver_port=0
--tserver_master_addrs=127.7.46.190:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.46.131
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:50.252879 27421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:50.254153 27421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:01:50.263336 27421 server_base.cc:1034] running on GCE node
W20260620 15:01:50.263293 27426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:50.263291 27427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:50.263321 27429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:50.264076 27421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:50.264689 27421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:50.265869 27421 hybrid_clock.cc:648] HybridClock initialized: now 1781967710265849 us; error 31 us; skew 500 ppm
I20260620 15:01:50.267573 27421 webserver.cc:460] Webserver started at http://127.7.46.131:45573/ using document root <none> and password file <none>
I20260620 15:01:50.268167 27421 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:50.268255 27421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:50.268472 27421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:50.270330 27421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/data/instance:
uuid: "1b1648c8fb774efa873a42b5ac806d08"
format_stamp: "Formatted at 2026-06-20 15:01:50 on dist-test-slave-30t4"
server_key: "ca4d5ed1ad1558dd62781bdfcd709d9e"
server_key_iv: "99ecda8c4ab4ff69aadb6d7600730306"
server_key_version: "encryptionkey@0"
I20260620 15:01:50.270856 27421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance:
uuid: "1b1648c8fb774efa873a42b5ac806d08"
format_stamp: "Formatted at 2026-06-20 15:01:50 on dist-test-slave-30t4"
server_key: "ca4d5ed1ad1558dd62781bdfcd709d9e"
server_key_iv: "99ecda8c4ab4ff69aadb6d7600730306"
server_key_version: "encryptionkey@0"
I20260620 15:01:50.274230 27421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 15:01:50.276388 27435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:50.277474 27421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:50.277596 27421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/wal
uuid: "1b1648c8fb774efa873a42b5ac806d08"
format_stamp: "Formatted at 2026-06-20 15:01:50 on dist-test-slave-30t4"
server_key: "ca4d5ed1ad1558dd62781bdfcd709d9e"
server_key_iv: "99ecda8c4ab4ff69aadb6d7600730306"
server_key_version: "encryptionkey@0"
I20260620 15:01:50.277696 27421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:50.289700 27421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:50.290268 27421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:50.290447 27421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:50.290946 27421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:50.291757 27421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:50.291887 27421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:50.291981 27421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:50.292027 27421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:50.300237 27421 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.131:33089
I20260620 15:01:50.300272 27548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.131:33089 every 8 connection(s)
I20260620 15:01:50.301023 27421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:50.307629 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 27421
I20260620 15:01:50.307749 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:50.308074 7354 external_mini_cluster.cc:1428] Setting key e06774fb873f72f7485231f5e75ab7b4
I20260620 15:01:50.310122 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.302726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:46191 (local address 127.7.46.190:43537)
0620 15:01:50.302854 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.302858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.303768 (+ 910us) server_negotiation.cc:407] Connection header received
0620 15:01:50.305197 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.305200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.305261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.305355 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:50.306211 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.306724 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.307883 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.308236 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.308350 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.309603 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:50.309606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:50.309608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:50.309618 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:50.309636 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:50.309643 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:50.309705 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.309869 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.309948 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:01:50.310842 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.303061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:43537 (local address 127.7.46.131:46191)
0620 15:01:50.303635 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:50.303670 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:50.304998 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:50.305509 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:50.305515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:50.305590 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:50.306078 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.306091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.306843 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.306848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.307672 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.307681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.308417 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.308422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.308622 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.309301 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:50.309303 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:50.309359 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:50.309457 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:50.309727 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:50.309734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:50.309740 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:01:50.309863 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 15:01:50.310119 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260620 15:01:50.311861 27549 heartbeater.cc:347] Connected to a master server at 127.7.46.190:43537
I20260620 15:01:50.312093 27549 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:50.312558 27549 heartbeater.cc:511] Master 127.7.46.190:43537 requested a full tablet report, sending...
I20260620 15:01:50.313470 27098 ts_manager.cc:194] Registered new tserver with Master: 1b1648c8fb774efa873a42b5ac806d08 (127.7.46.131:33089)
I20260620 15:01:50.313983 27098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.46.131:46191
I20260620 15:01:50.321573 7354 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:01:50.327262 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.323433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.7.46.190:43537)
0620 15:01:50.323592 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.323597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.323731 (+ 134us) server_negotiation.cc:407] Connection header received
0620 15:01:50.323898 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.323901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.323943 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.324042 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:50.324311 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.324830 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.325662 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.325898 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.326014 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.326817 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:50.326821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:50.326824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:50.326837 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:50.326857 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:50.326874 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:50.326955 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.327067 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.327122 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:01:50.330361 27098 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:01:50.333448 27098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:01:50.346872 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.341734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.7.46.131:33089)
0620 15:01:50.342111 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.342124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.342152 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:01:50.342232 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.342237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.342447 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.342607 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:50.343112 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.344238 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.345569 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.346089 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.346206 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.346406 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.346618 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.346711 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":160,"threads_started":1}
I20260620 15:01:50.346868 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.341559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.131:33089 (local address 127.0.0.1:44400)
0620 15:01:50.341903 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:50.341972 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:50.342098 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:50.342799 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:50.342805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:50.342828 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:50.342971 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.342981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.344336 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.344342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.345427 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.345436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.346216 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.346220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.346417 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.346473 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 15:01:50.346589 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 15:01:50.346706 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":56,"thread_start_us":110,"threads_started":1}
I20260620 15:01:50.348296 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.342520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.7.46.130:43055)
0620 15:01:50.342910 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.342922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.343241 (+ 319us) server_negotiation.cc:407] Connection header received
0620 15:01:50.343317 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.343322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.343524 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.343672 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:50.343965 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.345170 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.346864 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.347645 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.347750 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.347952 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.348030 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.348119 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":96,"threads_started":1}
I20260620 15:01:50.348398 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.342669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45959 (local address 127.0.0.1:49680)
0620 15:01:50.343134 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:50.343151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:50.343244 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:50.343896 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:50.343899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:50.343915 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:50.343999 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.344004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.345308 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.345312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.346722 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.346733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.348013 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.348017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.348102 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.348123 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:01:50.348193 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 15:01:50.348259 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":96,"thread_start_us":81,"threads_started":1}
I20260620 15:01:50.348645 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.342340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:43055 (local address 127.0.0.1:34028)
0620 15:01:50.343053 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:50.343067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:50.343150 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:50.343663 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:50.343666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:50.343684 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:50.343820 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.343831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.345308 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.345312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.346726 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.346733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.347798 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.347803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.347888 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.347910 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:01:50.347995 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:01:50.348527 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":217,"thread_start_us":101,"threads_started":1}
I20260620 15:01:50.348878 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.343068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.7.46.129:45959)
0620 15:01:50.343464 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.343472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.343490 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:01:50.343549 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.343552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.343734 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.343914 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:50.344082 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.345175 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.346864 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.347645 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.347750 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.347945 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.348602 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.348699 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":110,"threads_started":1}
I20260620 15:01:50.350402 27483 tablet_service.cc:1467] Processing CreateTablet for tablet 12b9cb949c4c43269d683b2a9b158d47 (DEFAULT_TABLE table=test-table [id=11c4a178f752424f9bcaf289bcd67aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:50.350869 27352 tablet_service.cc:1467] Processing CreateTablet for tablet 12b9cb949c4c43269d683b2a9b158d47 (DEFAULT_TABLE table=test-table [id=11c4a178f752424f9bcaf289bcd67aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:50.351511 27483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b9cb949c4c43269d683b2a9b158d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:50.351881 27352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b9cb949c4c43269d683b2a9b158d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:50.351733 27221 tablet_service.cc:1467] Processing CreateTablet for tablet 12b9cb949c4c43269d683b2a9b158d47 (DEFAULT_TABLE table=test-table [id=11c4a178f752424f9bcaf289bcd67aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:50.352679 27221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b9cb949c4c43269d683b2a9b158d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:50.357663 27566 tablet_bootstrap.cc:492] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Bootstrap starting.
I20260620 15:01:50.359009 27567 tablet_bootstrap.cc:492] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Bootstrap starting.
I20260620 15:01:50.359480 27568 tablet_bootstrap.cc:492] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78: Bootstrap starting.
I20260620 15:01:50.359997 27566 tablet_bootstrap.cc:654] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:50.360700 27566 log.cc:824] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:50.361127 27567 tablet_bootstrap.cc:654] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:50.361444 27568 tablet_bootstrap.cc:654] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:50.361954 27567 log.cc:824] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:50.362160 27568 log.cc:824] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:50.363209 27566 tablet_bootstrap.cc:492] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: No bootstrap required, opened a new log
I20260620 15:01:50.363410 27566 ts_tablet_manager.cc:1397] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 15:01:50.363735 27567 tablet_bootstrap.cc:492] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: No bootstrap required, opened a new log
I20260620 15:01:50.363936 27567 ts_tablet_manager.cc:1397] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 15:01:50.364408 27568 tablet_bootstrap.cc:492] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78: No bootstrap required, opened a new log
I20260620 15:01:50.364584 27568 ts_tablet_manager.cc:1397] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 15:01:50.366474 27566 raft_consensus.cc:348] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.366770 27566 raft_consensus.cc:374] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:50.366833 27566 raft_consensus.cc:729] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536ee079a473406593e7e63947481d6b, State: Initialized, Role: FOLLOWER
I20260620 15:01:50.367339 27566 consensus_queue.cc:260] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.368155 27418 heartbeater.cc:503] Master 127.7.46.190:43537 was elected leader, sending a full tablet report...
I20260620 15:01:50.368088 27567 raft_consensus.cc:348] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.368317 27567 raft_consensus.cc:374] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:50.368364 27567 raft_consensus.cc:729] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1648c8fb774efa873a42b5ac806d08, State: Initialized, Role: FOLLOWER
I20260620 15:01:50.368412 27566 ts_tablet_manager.cc:1428] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 15:01:50.368564 27568 raft_consensus.cc:348] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.368801 27568 raft_consensus.cc:374] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:50.368844 27568 raft_consensus.cc:729] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c01162261504690ad168fe64e52df78, State: Initialized, Role: FOLLOWER
I20260620 15:01:50.368794 27567 consensus_queue.cc:260] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.369226 27568 consensus_queue.cc:260] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.369683 27567 ts_tablet_manager.cc:1428] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 15:01:50.369973 27549 heartbeater.cc:503] Master 127.7.46.190:43537 was elected leader, sending a full tablet report...
I20260620 15:01:50.370071 27568 ts_tablet_manager.cc:1428] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 15:01:50.370393 27287 heartbeater.cc:503] Master 127.7.46.190:43537 was elected leader, sending a full tablet report...
W20260620 15:01:50.421891 27288 tablet.cc:2369] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:01:50.552018 27550 tablet.cc:2369] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:01:50.611333 27419 tablet.cc:2369] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:50.670104 27573 raft_consensus.cc:482] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:50.670292 27573 raft_consensus.cc:504] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.671315 27573 leader_election.cc:288] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 536ee079a473406593e7e63947481d6b (127.7.46.130:43055), 2c01162261504690ad168fe64e52df78 (127.7.46.129:45959)
I20260620 15:01:50.674942 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.671651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:60837 (local address 127.7.46.130:43055)
0620 15:01:50.671811 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.671816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.671833 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:01:50.671978 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.671984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.672037 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.672121 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:50.672389 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.673014 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.674024 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.674412 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.674545 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.674595 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.674764 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.674829 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:01:50.674952 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.671581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.130:43055 (local address 127.7.46.131:60837)
0620 15:01:50.671720 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:50.671737 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:50.671869 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:50.672143 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:50.672146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:50.672165 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:50.672281 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.672287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.673127 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.673131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.673895 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.673903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.674545 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.674549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.674649 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.674666 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 15:01:50.674750 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 15:01:50.674815 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 15:01:50.675400 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.672050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.46.131:42093 (local address 127.7.46.129:45959)
0620 15:01:50.672179 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.672185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.672411 (+ 226us) server_negotiation.cc:407] Connection header received
0620 15:01:50.672565 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.672568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.672619 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.672705 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:50.672970 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.673448 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.674464 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.674869 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.675006 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.675127 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.675234 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.675298 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:01:50.675415 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.671920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45959 (local address 127.7.46.131:42093)
0620 15:01:50.672319 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:50.672330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:50.672446 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:50.672717 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:50.672719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:50.672735 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:50.672863 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.672868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.673571 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.673574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.674333 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:50.674341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.675018 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.675022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:50.675117 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.675141 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:01:50.675219 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 15:01:50.675280 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":122,"threads_started":1}
I20260620 15:01:50.675601 27372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b9cb949c4c43269d683b2a9b158d47" candidate_uuid: "1b1648c8fb774efa873a42b5ac806d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536ee079a473406593e7e63947481d6b" is_pre_election: true
I20260620 15:01:50.675932 27372 raft_consensus.cc:2454] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b1648c8fb774efa873a42b5ac806d08 in term 0.
I20260620 15:01:50.676254 27241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b9cb949c4c43269d683b2a9b158d47" candidate_uuid: "1b1648c8fb774efa873a42b5ac806d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c01162261504690ad168fe64e52df78" is_pre_election: true
I20260620 15:01:50.676456 27436 leader_election.cc:302] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1648c8fb774efa873a42b5ac806d08, 536ee079a473406593e7e63947481d6b; no voters:
I20260620 15:01:50.676519 27241 raft_consensus.cc:2454] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b1648c8fb774efa873a42b5ac806d08 in term 0.
I20260620 15:01:50.676728 27573 raft_consensus.cc:2781] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:01:50.676820 27573 raft_consensus.cc:482] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:01:50.676878 27573 raft_consensus.cc:3037] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:50.677827 27573 raft_consensus.cc:504] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.678153 27573 leader_election.cc:288] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [CANDIDATE]: Term 1 election: Requested vote from peers 536ee079a473406593e7e63947481d6b (127.7.46.130:43055), 2c01162261504690ad168fe64e52df78 (127.7.46.129:45959)
I20260620 15:01:50.678540 27241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b9cb949c4c43269d683b2a9b158d47" candidate_uuid: "1b1648c8fb774efa873a42b5ac806d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c01162261504690ad168fe64e52df78"
I20260620 15:01:50.678603 27372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b9cb949c4c43269d683b2a9b158d47" candidate_uuid: "1b1648c8fb774efa873a42b5ac806d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536ee079a473406593e7e63947481d6b"
I20260620 15:01:50.678653 27241 raft_consensus.cc:3037] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:50.678742 27372 raft_consensus.cc:3037] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:50.679653 27372 raft_consensus.cc:2454] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b1648c8fb774efa873a42b5ac806d08 in term 1.
I20260620 15:01:50.679662 27241 raft_consensus.cc:2454] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b1648c8fb774efa873a42b5ac806d08 in term 1.
I20260620 15:01:50.679968 27436 leader_election.cc:302] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1648c8fb774efa873a42b5ac806d08, 536ee079a473406593e7e63947481d6b; no voters:
I20260620 15:01:50.680163 27573 raft_consensus.cc:2781] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:50.680366 27573 raft_consensus.cc:686] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 1 LEADER]: Becoming Leader. State: Replica: 1b1648c8fb774efa873a42b5ac806d08, State: Running, Role: LEADER
I20260620 15:01:50.680687 27573 consensus_queue.cc:237] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } }
I20260620 15:01:50.683933 27098 catalog_manager.cc:5495] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b1648c8fb774efa873a42b5ac806d08 (127.7.46.131). New cstate: current_term: 1 leader_uuid: "1b1648c8fb774efa873a42b5ac806d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1648c8fb774efa873a42b5ac806d08" member_type: VOTER last_known_addr { host: "127.7.46.131" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 } health_report { overall_health: UNKNOWN } } }
I20260620 15:01:50.705982 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.702677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.7.46.131:33089)
0620 15:01:50.702806 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:50.702810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:50.702823 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:01:50.702951 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:50.702954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:50.703006 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:50.703088 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:50.703280 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.703754 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.704609 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:50.704815 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:50.704921 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:50.705206 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:50.705623 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:01:50.705739 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:50.705799 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 15:01:50.705847 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:01:50.711534 27241 raft_consensus.cc:1261] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 1 FOLLOWER]: Refusing update from remote peer 1b1648c8fb774efa873a42b5ac806d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:50.711534 27372 raft_consensus.cc:1261] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 1 FOLLOWER]: Refusing update from remote peer 1b1648c8fb774efa873a42b5ac806d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:50.712234 27573 consensus_queue.cc:1035] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c01162261504690ad168fe64e52df78" member_type: VOTER last_known_addr { host: "127.7.46.129" port: 45959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:50.712440 27576 consensus_queue.cc:1035] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "536ee079a473406593e7e63947481d6b" member_type: VOTER last_known_addr { host: "127.7.46.130" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:50.719242 27579 mvcc.cc:205] Tried to move back new op lower bound from 7298939743066988544 to 7298939742950645760. Current Snapshot: MvccSnapshot[applied={T|T < 7298939743066988544}]
I20260620 15:01:50.721047 27580 mvcc.cc:205] Tried to move back new op lower bound from 7298939743066988544 to 7298939742950645760. Current Snapshot: MvccSnapshot[applied={T|T < 7298939743066988544}]
I20260620 15:01:50.728026 27098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:50.728175 27098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:50.730264 27098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63cb90db25bc4ad294ddccd392f9ffef: Sending DeleteTablet for 3 replicas of tablet 12b9cb949c4c43269d683b2a9b158d47
I20260620 15:01:50.731038 27483 tablet_service.cc:1514] Processing DeleteTablet for tablet 12b9cb949c4c43269d683b2a9b158d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:50 UTC) from {username='slave'} at 127.0.0.1:44400
I20260620 15:01:50.731038 27221 tablet_service.cc:1514] Processing DeleteTablet for tablet 12b9cb949c4c43269d683b2a9b158d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:50 UTC) from {username='slave'} at 127.0.0.1:49680
I20260620 15:01:50.731504 27590 tablet_replica.cc:330] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78: stopping tablet replica
I20260620 15:01:50.731487 27352 tablet_service.cc:1514] Processing DeleteTablet for tablet 12b9cb949c4c43269d683b2a9b158d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:50 UTC) from {username='slave'} at 127.0.0.1:34028
I20260620 15:01:50.731828 27590 raft_consensus.cc:2229] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:50.731963 27591 tablet_replica.cc:330] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: stopping tablet replica
I20260620 15:01:50.732137 27590 raft_consensus.cc:2258] T 12b9cb949c4c43269d683b2a9b158d47 P 2c01162261504690ad168fe64e52df78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:50.732317 27591 raft_consensus.cc:2229] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:50.732378 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 27159
I20260620 15:01:50.732496 27589 tablet_replica.cc:330] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: stopping tablet replica
I20260620 15:01:50.732746 27589 raft_consensus.cc:2229] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:01:50.733352 27589 raft_consensus.cc:2258] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:50.733961 27591 raft_consensus.cc:2258] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:50.735355 27591 ts_tablet_manager.cc:1905] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:50.737418 27591 ts_tablet_manager.cc:1918] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:50.737526 27591 log.cc:1196] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-1/wal/wals/12b9cb949c4c43269d683b2a9b158d47
I20260620 15:01:50.737844 27591 ts_tablet_manager.cc:1939] T 12b9cb949c4c43269d683b2a9b158d47 P 536ee079a473406593e7e63947481d6b: Deleting consensus metadata
I20260620 15:01:50.738772 27589 ts_tablet_manager.cc:1905] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:50.738991 27083 catalog_manager.cc:4841] TS 536ee079a473406593e7e63947481d6b (127.7.46.130:43055): tablet 12b9cb949c4c43269d683b2a9b158d47 (table test-table [id=11c4a178f752424f9bcaf289bcd67aa9]) successfully deleted
W20260620 15:01:50.740267 27083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.46.129:45959 (error 108)
I20260620 15:01:50.741214 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:50.740922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.129:45959 (local address 127.0.0.1:49690)
0620 15:01:50.741056 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:50.741129 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.129:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:01:50.741261 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 27290
I20260620 15:01:50.741434 27589 ts_tablet_manager.cc:1918] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:50.741525 27589 log.cc:1196] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Deleting WAL directory at /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967464421185-7354-0/minicluster-data/ts-2/wal/wals/12b9cb949c4c43269d683b2a9b158d47
I20260620 15:01:50.741796 27589 ts_tablet_manager.cc:1939] T 12b9cb949c4c43269d683b2a9b158d47 P 1b1648c8fb774efa873a42b5ac806d08: Deleting consensus metadata
W20260620 15:01:50.743323 27083 catalog_manager.cc:4584] TS 2c01162261504690ad168fe64e52df78 (127.7.46.129:45959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12b9cb949c4c43269d683b2a9b158d47: Network error: Client connection negotiation failed: client connection to 127.7.46.129:45959: connect: Connection refused (error 111)
I20260620 15:01:50.743537 27083 catalog_manager.cc:4841] TS 1b1648c8fb774efa873a42b5ac806d08 (127.7.46.131:33089): tablet 12b9cb949c4c43269d683b2a9b158d47 (table test-table [id=11c4a178f752424f9bcaf289bcd67aa9]) successfully deleted
I20260620 15:01:50.749074 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 27421
I20260620 15:01:50.754812 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 27068
2026-06-20T15:01:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1265 ms)
[----------] 6 tests from EncryptionPolicyTest (9941 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:50 dist-test-slave-30t4 krb5kdc[27595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:50 dist-test-slave-30t4 krb5kdc[27595](info): set up 2 sockets
Jun 20 15:01:50 dist-test-slave-30t4 krb5kdc[27595](info): commencing operation
krb5kdc: starting...
W20260620 15:01:51.799899 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:51 dist-test-slave-30t4 krb5kdc[27595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:01:51.917007 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:52.022200 27608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:52.022465 27608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:52.022548 27608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:52.026506 27608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:52.026588 27608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:52.026626 27608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:52.026682 27608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:52.026722 27608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:52.033049 27608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:52.034011 27608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:52.034970 27608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:52.043146 27616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:52.043184 27613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:52.043174 27614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:52.043313 27608 server_base.cc:1034] running on GCE node
I20260620 15:01:52.043977 27608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:52.044989 27608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:52.046171 27608 hybrid_clock.cc:648] HybridClock initialized: now 1781967712046157 us; error 26 us; skew 500 ppm
Jun 20 15:01:52 dist-test-slave-30t4 krb5kdc[27595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:52.048777 27608 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:01:52.049585 27608 webserver.cc:460] Webserver started at http://127.0.0.1:38115/ using document root <none> and password file <none>
I20260620 15:01:52.050074 27608 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:52.050132 27608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:52.050282 27608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:52.052181 27608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "18d002845687415c826b42b0c029a210"
format_stamp: "Formatted at 2026-06-20 15:01:52 on dist-test-slave-30t4"
server_key: "4ebc1ffc0dccc58e3875e9e5403e3fcf"
server_key_iv: "3bcf33133b5d92fe4de26a686d4f18c0"
server_key_version: "encryptionkey@0"
I20260620 15:01:52.052688 27608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "18d002845687415c826b42b0c029a210"
format_stamp: "Formatted at 2026-06-20 15:01:52 on dist-test-slave-30t4"
server_key: "4ebc1ffc0dccc58e3875e9e5403e3fcf"
server_key_iv: "3bcf33133b5d92fe4de26a686d4f18c0"
server_key_version: "encryptionkey@0"
I20260620 15:01:52.056136 27608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:52.058312 27623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:52.059414 27608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:52.059520 27608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "18d002845687415c826b42b0c029a210"
format_stamp: "Formatted at 2026-06-20 15:01:52 on dist-test-slave-30t4"
server_key: "4ebc1ffc0dccc58e3875e9e5403e3fcf"
server_key_iv: "3bcf33133b5d92fe4de26a686d4f18c0"
server_key_version: "encryptionkey@0"
I20260620 15:01:52.059598 27608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:52.089136 27608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:52.091924 27608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:52.092063 27608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:52.099100 27608 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38005
I20260620 15:01:52.099125 27675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38005 every 8 connection(s)
I20260620 15:01:52.100008 27608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:52.102442 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 27608
I20260620 15:01:52.102576 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:52.102674 27676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:52.102895 7354 external_mini_cluster.cc:1428] Setting key 649635d627e6efa4125fc3cf6a1415e5
I20260620 15:01:52.108625 27676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210: Bootstrap starting.
I20260620 15:01:52.110673 27676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210: Neither blocks nor log segments found. Creating new log.
Jun 20 15:01:52 dist-test-slave-30t4 krb5kdc[27595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:01:52.111397 27676 log.cc:824] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:52.113124 27676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210: No bootstrap required, opened a new log
I20260620 15:01:52.116425 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:52.104252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45955 (local address 127.0.0.1:38005)
0620 15:01:52.104673 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:52.104687 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:52.104721 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:01:52.105330 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:52.105345 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:52.105658 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:52.105985 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:52.106821 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:52.107808 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:52.108665 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:52.109189 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:52.109352 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:52.111836 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:52.111841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:52.111852 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:52.111878 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:52.113613 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:52.114111 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:52.114116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:52.114120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:52.114235 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:52.114575 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:52.114581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:52.114588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:52.114957 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:52.115169 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:52.115480 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 15:01:52.115683 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":142,"threads_started":1}
I20260620 15:01:52.116498 27676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d002845687415c826b42b0c029a210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260620 15:01:52.116772 27676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:52.116851 27676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d002845687415c826b42b0c029a210, State: Initialized, Role: FOLLOWER
I20260620 15:01:52.117249 27676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d002845687415c826b42b0c029a210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260620 15:01:52.117398 27676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:52.117465 27676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:52.117554 27676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:52.118356 27676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d002845687415c826b42b0c029a210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260620 15:01:52.118708 27676 leader_election.cc:302] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18d002845687415c826b42b0c029a210; no voters:
I20260620 15:01:52.119019 27676 leader_election.cc:288] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:52.119118 27681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:52.119324 27681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [term 1 LEADER]: Becoming Leader. State: Replica: 18d002845687415c826b42b0c029a210, State: Running, Role: LEADER
I20260620 15:01:52.119374 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:01:52.119613 27681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d002845687415c826b42b0c029a210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260620 15:01:52.120218 27676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:52.120823 27682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d002845687415c826b42b0c029a210. Latest consensus state: current_term: 1 leader_uuid: "18d002845687415c826b42b0c029a210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d002845687415c826b42b0c029a210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } } }
I20260620 15:01:52.120957 27682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:52.121611 27683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18d002845687415c826b42b0c029a210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d002845687415c826b42b0c029a210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } } }
I20260620 15:01:52.121699 27683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210 [sys.catalog]: This master's current role is: LEADER
W20260620 15:01:52.125082 27701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:01:52.125145 27701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:01:52.125224 27692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:52.126071 27692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:52.130800 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:52.121589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58361 (local address 127.0.0.1:38005)
0620 15:01:52.121686 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:52.121689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:52.122842 (+ 1153us) server_negotiation.cc:407] Connection header received
0620 15:01:52.122960 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:52.122966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:52.123011 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:52.123089 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:52.123711 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:52.124453 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:52.125598 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:52.125809 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:52.125887 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:52.127374 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:52.127377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:52.127378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:52.127398 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:52.128942 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:52.129498 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:52.129501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:52.129503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:52.129588 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:52.129916 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:52.129920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:52.129922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:52.130136 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:52.130280 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:52.130546 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 15:01:52.130631 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:01:52.132073 27692 catalog_manager.cc:1347] Generated new cluster ID: 0bdce79095a94a1784b3cd66982cdacd
I20260620 15:01:52.132139 27692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:52.142328 27692 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:52.143198 27692 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:52.150812 27692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d002845687415c826b42b0c029a210: Generated new TSK 0
I20260620 15:01:52.151400 27692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:52.157081 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 27608
W20260620 15:01:52.162401 27593 connection.cc:535] client connection to 127.0.0.1:38005 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38005): Connection reset by peer (error 104)
2026-06-20T15:01:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:52 dist-test-slave-30t4 krb5kdc[27707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:52 dist-test-slave-30t4 krb5kdc[27707](info): set up 2 sockets
Jun 20 15:01:52 dist-test-slave-30t4 krb5kdc[27707](info): commencing operation
krb5kdc: starting...
W20260620 15:01:53.199847 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:53 dist-test-slave-30t4 krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:01:53.312162 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:53.416404 27720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:53.416642 27720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:53.416683 27720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:53.420599 27720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:53.420650 27720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:53.420670 27720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:53.420686 27720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:53.420732 27720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:53.425800 27720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:53.426741 27720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:53.427728 27720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:53.435702 27725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:53.435717 27726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:53.435716 27728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:53.436090 27720 server_base.cc:1034] running on GCE node
I20260620 15:01:53.436602 27720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:53.437613 27720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:53.438789 27720 hybrid_clock.cc:648] HybridClock initialized: now 1781967713438761 us; error 39 us; skew 500 ppm
Jun 20 15:01:53 dist-test-slave-30t4 krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:53.441520 27720 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:01:53.442400 27720 webserver.cc:460] Webserver started at http://127.0.0.1:40609/ using document root <none> and password file <none>
I20260620 15:01:53.442945 27720 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:53.443003 27720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:53.443205 27720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:53.445189 27720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "e829e826e80d48bfa13423154f1a5391"
format_stamp: "Formatted at 2026-06-20 15:01:53 on dist-test-slave-30t4"
server_key: "2e5cf4444ca30e825dff5b6786163a93"
server_key_iv: "f5fd173268b4e14ad76c50955797dd68"
server_key_version: "encryptionkey@0"
I20260620 15:01:53.445794 27720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "e829e826e80d48bfa13423154f1a5391"
format_stamp: "Formatted at 2026-06-20 15:01:53 on dist-test-slave-30t4"
server_key: "2e5cf4444ca30e825dff5b6786163a93"
server_key_iv: "f5fd173268b4e14ad76c50955797dd68"
server_key_version: "encryptionkey@0"
I20260620 15:01:53.449323 27720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:01:53.451550 27735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:53.452695 27720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:53.452837 27720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "e829e826e80d48bfa13423154f1a5391"
format_stamp: "Formatted at 2026-06-20 15:01:53 on dist-test-slave-30t4"
server_key: "2e5cf4444ca30e825dff5b6786163a93"
server_key_iv: "f5fd173268b4e14ad76c50955797dd68"
server_key_version: "encryptionkey@0"
I20260620 15:01:53.452943 27720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:53.461360 27720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:53.464167 27720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:53.464360 27720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:53.470875 27720 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37419
I20260620 15:01:53.470886 27787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37419 every 8 connection(s)
I20260620 15:01:53.471729 27720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:53.474277 27788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:53.477134 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 27720
I20260620 15:01:53.477316 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:53.477632 7354 external_mini_cluster.cc:1428] Setting key 0476de6e668924a877d5714dac3c10b9
I20260620 15:01:53.480000 27788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391: Bootstrap starting.
I20260620 15:01:53.482514 27788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:53.483284 27788 log.cc:824] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:53.485173 27788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391: No bootstrap required, opened a new log
I20260620 15:01:53.487003 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:53.478950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49807 (local address 127.0.0.1:37419)
0620 15:01:53.479333 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:53.479352 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:53.479391 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:01:53.480230 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:53.480252 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:53.480659 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:53.480995 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:53.481399 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:53.482882 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:53.483827 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:53.484287 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:53.484465 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:53.485321 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:53.485325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:53.485336 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:53.485678 (+ 342us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:53.485726 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:53.485747 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:53.485883 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:53.486057 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 15:01:53.486184 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":98,"threads_started":1}
I20260620 15:01:53.488406 27788 raft_consensus.cc:348] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e829e826e80d48bfa13423154f1a5391" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } }
I20260620 15:01:53.488654 27788 raft_consensus.cc:374] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:53.488723 27788 raft_consensus.cc:729] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e829e826e80d48bfa13423154f1a5391, State: Initialized, Role: FOLLOWER
I20260620 15:01:53.489192 27788 consensus_queue.cc:260] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e829e826e80d48bfa13423154f1a5391" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } }
I20260620 15:01:53.489355 27788 raft_consensus.cc:388] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:53.489426 27788 raft_consensus.cc:482] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:53.489521 27788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:53.489991 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:01:53.490478 27788 raft_consensus.cc:504] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e829e826e80d48bfa13423154f1a5391" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } }
I20260620 15:01:53.490795 27788 leader_election.cc:302] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e829e826e80d48bfa13423154f1a5391; no voters:
I20260620 15:01:53.491094 27788 leader_election.cc:288] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:53.491154 27794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:53.491344 27794 raft_consensus.cc:686] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [term 1 LEADER]: Becoming Leader. State: Replica: e829e826e80d48bfa13423154f1a5391, State: Running, Role: LEADER
I20260620 15:01:53.491684 27794 consensus_queue.cc:237] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e829e826e80d48bfa13423154f1a5391" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } }
I20260620 15:01:53.492324 27788 sys_catalog.cc:564] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:53.493327 27798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e829e826e80d48bfa13423154f1a5391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e829e826e80d48bfa13423154f1a5391" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } } }
I20260620 15:01:53.493459 27798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:53.493798 27799 sys_catalog.cc:455] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e829e826e80d48bfa13423154f1a5391. Latest consensus state: current_term: 1 leader_uuid: "e829e826e80d48bfa13423154f1a5391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e829e826e80d48bfa13423154f1a5391" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } } }
I20260620 15:01:53.493947 27799 sys_catalog.cc:458] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:53.494493 27807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:53.496799 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:53.492009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53063 (local address 127.0.0.1:37419)
0620 15:01:53.492108 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:53.492111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:53.492369 (+ 258us) server_negotiation.cc:407] Connection header received
0620 15:01:53.492509 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:53.492512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:53.492593 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:53.492656 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:53.493041 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:53.493803 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:53.494905 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:53.495182 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:53.495288 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:53.496319 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:53.496323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:53.496326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:53.496339 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:53.496367 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:53.496382 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:53.496471 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:53.496643 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 15:01:53.496676 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 15:01:53.497740 27807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:53.502796 27807 catalog_manager.cc:1347] Generated new cluster ID: f440d45ff51f434299d76db6a12f0ac0
I20260620 15:01:53.502876 27807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:53.514117 27807 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:53.514973 27807 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:53.521001 27807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e829e826e80d48bfa13423154f1a5391: Generated new TSK 0
I20260620 15:01:53.521674 27807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:53.531565 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 27720
W20260620 15:01:53.536988 27705 connection.cc:535] client connection to 127.0.0.1:37419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37419): Connection reset by peer (error 104)
2026-06-20T15:01:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:53 dist-test-slave-30t4 krb5kdc[27819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:53 dist-test-slave-30t4 krb5kdc[27819](info): set up 2 sockets
Jun 20 15:01:53 dist-test-slave-30t4 krb5kdc[27819](info): commencing operation
krb5kdc: starting...
W20260620 15:01:54.563913 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:54 dist-test-slave-30t4 krb5kdc[27819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:01:54.677886 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:54.784897 27832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:54.785151 27832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:54.785220 27832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:54.789083 27832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:54.789160 27832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:54.789199 27832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:54.789252 27832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:54.789288 27832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:54.794346 27832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:54.795266 27832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:54.796221 27832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:54.804149 27837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:54.804149 27840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:54.804158 27838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:54.804286 27832 server_base.cc:1034] running on GCE node
I20260620 15:01:54.804934 27832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:54.805905 27832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:54.807085 27832 hybrid_clock.cc:648] HybridClock initialized: now 1781967714807070 us; error 24 us; skew 500 ppm
Jun 20 15:01:54 dist-test-slave-30t4 krb5kdc[27819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:54.809702 27832 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:01:54.810516 27832 webserver.cc:460] Webserver started at http://127.0.0.1:39049/ using document root <none> and password file <none>
I20260620 15:01:54.810992 27832 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:54.811049 27832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:54.811199 27832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:54.813107 27832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "329ab2f36d334b45b88fe6c1d1954237"
format_stamp: "Formatted at 2026-06-20 15:01:54 on dist-test-slave-30t4"
server_key: "5fea7ab0487647ef45e9aa9ce94b0021"
server_key_iv: "ad607c503b4aa7d5d2ff7b41594569c2"
server_key_version: "encryptionkey@0"
I20260620 15:01:54.813616 27832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "329ab2f36d334b45b88fe6c1d1954237"
format_stamp: "Formatted at 2026-06-20 15:01:54 on dist-test-slave-30t4"
server_key: "5fea7ab0487647ef45e9aa9ce94b0021"
server_key_iv: "ad607c503b4aa7d5d2ff7b41594569c2"
server_key_version: "encryptionkey@0"
I20260620 15:01:54.817135 27832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 15:01:54.819420 27847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:54.820612 27832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:01:54.820715 27832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "329ab2f36d334b45b88fe6c1d1954237"
format_stamp: "Formatted at 2026-06-20 15:01:54 on dist-test-slave-30t4"
server_key: "5fea7ab0487647ef45e9aa9ce94b0021"
server_key_iv: "ad607c503b4aa7d5d2ff7b41594569c2"
server_key_version: "encryptionkey@0"
I20260620 15:01:54.820812 27832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:54.825019 27832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:54.828205 27832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:54.828341 27832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:54.835011 27832 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37271
I20260620 15:01:54.835021 27899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37271 every 8 connection(s)
I20260620 15:01:54.835914 27832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:54.838469 27900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:54.842691 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 27832
I20260620 15:01:54.842836 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:54.843164 7354 external_mini_cluster.cc:1428] Setting key 75c0509a625c6dc56fc380b6c3612a0b
I20260620 15:01:54.844002 27900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237: Bootstrap starting.
I20260620 15:01:54.846248 27900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:54.847180 27900 log.cc:824] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:54.848112 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:54.844421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54671 (local address 127.0.0.1:37271)
0620 15:01:54.844830 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:54.844843 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:54.844881 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:01:54.845470 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:54.845486 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:54.845760 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:54.846018 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:54.846792 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:54.846797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:54.846808 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:54.846817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:54.846865 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:54.846883 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:54.846966 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:54.847114 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:01:54.847280 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
I20260620 15:01:54.849040 27900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237: No bootstrap required, opened a new log
I20260620 15:01:54.850871 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:01:54.853440 27900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329ab2f36d334b45b88fe6c1d1954237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260620 15:01:54.853704 27900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:54.853773 27900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329ab2f36d334b45b88fe6c1d1954237, State: Initialized, Role: FOLLOWER
I20260620 15:01:54.854185 27900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329ab2f36d334b45b88fe6c1d1954237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260620 15:01:54.854331 27900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:54.854398 27900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:54.854497 27900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:54.855069 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:54.853238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33977 (local address 127.0.0.1:37271)
0620 15:01:54.853376 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:54.853379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:54.853494 (+ 115us) server_negotiation.cc:407] Connection header received
0620 15:01:54.853641 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:54.853644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:54.853673 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:54.853790 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:54.854660 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:54.854664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:54.854666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:54.854678 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:54.854704 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:54.854720 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:54.854797 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:54.854918 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 15:01:54.854947 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:01:54.855794 27900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329ab2f36d334b45b88fe6c1d1954237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260620 15:01:54.856159 27900 leader_election.cc:302] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 329ab2f36d334b45b88fe6c1d1954237; no voters:
I20260620 15:01:54.856412 27900 leader_election.cc:288] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:54.856532 27910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:54.856760 27910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [term 1 LEADER]: Becoming Leader. State: Replica: 329ab2f36d334b45b88fe6c1d1954237, State: Running, Role: LEADER
I20260620 15:01:54.857090 27910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329ab2f36d334b45b88fe6c1d1954237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260620 15:01:54.857547 27900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:54.858745 27912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 329ab2f36d334b45b88fe6c1d1954237. Latest consensus state: current_term: 1 leader_uuid: "329ab2f36d334b45b88fe6c1d1954237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329ab2f36d334b45b88fe6c1d1954237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260620 15:01:54.858881 27912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:54.859153 27911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "329ab2f36d334b45b88fe6c1d1954237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329ab2f36d334b45b88fe6c1d1954237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260620 15:01:54.859246 27911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:54.860046 27919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:54.862653 27919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:54.868162 27919 catalog_manager.cc:1347] Generated new cluster ID: e9c78f5fa56b4baba3334547e7d5b3a7
I20260620 15:01:54.868227 27919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:54.895032 27919 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:54.895325 27919 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:54.899236 27919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 329ab2f36d334b45b88fe6c1d1954237: Generated new TSK 0
I20260620 15:01:54.899901 27919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:54.915021 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 27832
2026-06-20T15:01:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:54 dist-test-slave-30t4 krb5kdc[27931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:54 dist-test-slave-30t4 krb5kdc[27931](info): set up 2 sockets
Jun 20 15:01:54 dist-test-slave-30t4 krb5kdc[27931](info): commencing operation
krb5kdc: starting...
W20260620 15:01:55.947949 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:56 dist-test-slave-30t4 krb5kdc[27931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:01:56.064978 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:56.170439 27944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:56.170708 27944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:56.170770 27944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:56.174902 27944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:56.175038 27944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:56.175130 27944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:56.175172 27944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:56.175218 27944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:56.180482 27944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:56.181466 27944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:56.182444 27944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:56.190207 27949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:56.190230 27950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:56.190379 27944 server_base.cc:1034] running on GCE node
W20260620 15:01:56.190240 27952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:56.191011 27944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:56.192076 27944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:56.193235 27944 hybrid_clock.cc:648] HybridClock initialized: now 1781967716193220 us; error 26 us; skew 500 ppm
Jun 20 15:01:56 dist-test-slave-30t4 krb5kdc[27931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:56.202690 27944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:01:56.203497 27944 webserver.cc:460] Webserver started at http://127.0.0.1:45661/ using document root <none> and password file <none>
I20260620 15:01:56.204084 27944 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:56.204169 27944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:56.204383 27944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:56.206300 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be"
format_stamp: "Formatted at 2026-06-20 15:01:56 on dist-test-slave-30t4"
server_key: "391d97bacf6c120cefe5510dbc4a0975"
server_key_iv: "c020f30982aa0a8f7fef63f0bba8dcd7"
server_key_version: "encryptionkey@0"
I20260620 15:01:56.206844 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be"
format_stamp: "Formatted at 2026-06-20 15:01:56 on dist-test-slave-30t4"
server_key: "391d97bacf6c120cefe5510dbc4a0975"
server_key_iv: "c020f30982aa0a8f7fef63f0bba8dcd7"
server_key_version: "encryptionkey@0"
I20260620 15:01:56.210343 27944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 15:01:56.212687 27959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:56.213827 27944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:56.213968 27944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be"
format_stamp: "Formatted at 2026-06-20 15:01:56 on dist-test-slave-30t4"
server_key: "391d97bacf6c120cefe5510dbc4a0975"
server_key_iv: "c020f30982aa0a8f7fef63f0bba8dcd7"
server_key_version: "encryptionkey@0"
I20260620 15:01:56.214079 27944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:56.237617 27944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:56.240310 27944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:56.240502 27944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:56.247097 27944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45403
I20260620 15:01:56.247123 28011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45403 every 8 connection(s)
I20260620 15:01:56.247980 27944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:56.250142 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 27944
I20260620 15:01:56.250290 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:56.250568 7354 external_mini_cluster.cc:1428] Setting key 1337bd90e5463826c5cf7b279660235f
I20260620 15:01:56.250955 28012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:56.256979 28012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be: Bootstrap starting.
Jun 20 15:01:56 dist-test-slave-30t4 krb5kdc[27931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:01:56.259246 28012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:56.260107 28012 log.cc:824] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:56.262086 28012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be: No bootstrap required, opened a new log
I20260620 15:01:56.264194 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:56.251863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50281 (local address 127.0.0.1:45403)
0620 15:01:56.252270 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:56.252284 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:56.252318 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:01:56.253128 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:56.253145 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:56.253456 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:56.253788 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:56.254543 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:56.255548 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:56.256448 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:56.257229 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:56.257397 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:56.259913 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:56.259918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:56.259929 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:56.259955 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:56.261566 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:56.262023 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:56.262028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:56.262031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:56.262128 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:56.262421 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:56.262426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:56.262428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:56.262785 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:56.262974 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:56.263220 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 15:01:56.263480 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":137,"threads_started":1}
I20260620 15:01:56.265681 28012 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260620 15:01:56.265954 28012 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:56.266018 28012 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6d4f9f3c3f4f0d813782dbe6b475be, State: Initialized, Role: FOLLOWER
I20260620 15:01:56.266462 28012 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260620 15:01:56.266616 28012 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:56.266686 28012 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:56.266784 28012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:56.266825 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:01:56.267565 28012 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260620 15:01:56.267935 28012 leader_election.cc:302] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe6d4f9f3c3f4f0d813782dbe6b475be; no voters:
I20260620 15:01:56.268249 28012 leader_election.cc:288] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:56.268329 28021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:56.268507 28021 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [term 1 LEADER]: Becoming Leader. State: Replica: fe6d4f9f3c3f4f0d813782dbe6b475be, State: Running, Role: LEADER
I20260620 15:01:56.268795 28021 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260620 15:01:56.269416 28012 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:56.269899 28021 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe6d4f9f3c3f4f0d813782dbe6b475be. Latest consensus state: current_term: 1 leader_uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } } }
I20260620 15:01:56.270005 28021 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:56.270306 28022 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6d4f9f3c3f4f0d813782dbe6b475be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } } }
I20260620 15:01:56.270403 28022 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be [sys.catalog]: This master's current role is: LEADER
W20260620 15:01:56.274101 28037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:01:56.274173 28037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:01:56.274237 28027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:56.274853 28027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:56.276891 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:56.268959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56809 (local address 127.0.0.1:45403)
0620 15:01:56.269047 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:56.269051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:56.269258 (+ 207us) server_negotiation.cc:407] Connection header received
0620 15:01:56.269361 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:56.269364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:56.269401 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:56.269532 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:56.270271 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:56.271089 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:56.272154 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:56.272492 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:56.272604 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:56.273837 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:56.273841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:56.273843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:56.273866 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:56.275300 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:56.275686 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:56.275690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:56.275692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:56.275830 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:56.276098 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:56.276101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:56.276103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:56.276272 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:56.276397 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:56.276621 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 15:01:56.276731 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 15:01:56.279898 28027 catalog_manager.cc:1347] Generated new cluster ID: fd9a106e6b82451f8a30b108804feb5f
I20260620 15:01:56.279965 28027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:56.296566 28027 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:56.297650 28027 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:56.308986 28027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe6d4f9f3c3f4f0d813782dbe6b475be: Generated new TSK 0
I20260620 15:01:56.309664 28027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:56.313458 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 27944
2026-06-20T15:01:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:56 dist-test-slave-30t4 krb5kdc[28043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:56 dist-test-slave-30t4 krb5kdc[28043](info): set up 2 sockets
Jun 20 15:01:56 dist-test-slave-30t4 krb5kdc[28043](info): commencing operation
krb5kdc: starting...
W20260620 15:01:57.348032 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:57 dist-test-slave-30t4 krb5kdc[28043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:01:57.462134 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:57.567896 28056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:57.568145 28056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:57.568212 28056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:57.572098 28056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:57.572166 28056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:57.572202 28056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:57.572252 28056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:57.572290 28056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:57.577224 28056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:57.578117 28056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:57.579030 28056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:01:57.587160 28056 server_base.cc:1034] running on GCE node
W20260620 15:01:57.587078 28064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:57.587067 28061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:57.587054 28062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:57.587833 28056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:57.588763 28056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:57.589922 28056 hybrid_clock.cc:648] HybridClock initialized: now 1781967717589907 us; error 27 us; skew 500 ppm
Jun 20 15:01:57 dist-test-slave-30t4 krb5kdc[28043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:57.592485 28056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:01:57.593330 28056 webserver.cc:460] Webserver started at http://127.0.0.1:35639/ using document root <none> and password file <none>
I20260620 15:01:57.593847 28056 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:57.593927 28056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:57.594135 28056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:57.596129 28056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "208c6fd301de42439f2814500803a6c7"
format_stamp: "Formatted at 2026-06-20 15:01:57 on dist-test-slave-30t4"
server_key: "859a4cff7a2123eb5063f5873e5e34d3"
server_key_iv: "57a10922070afc36769a65cf5ea68e1e"
server_key_version: "encryptionkey@0"
I20260620 15:01:57.596658 28056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "208c6fd301de42439f2814500803a6c7"
format_stamp: "Formatted at 2026-06-20 15:01:57 on dist-test-slave-30t4"
server_key: "859a4cff7a2123eb5063f5873e5e34d3"
server_key_iv: "57a10922070afc36769a65cf5ea68e1e"
server_key_version: "encryptionkey@0"
I20260620 15:01:57.600255 28056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:57.602530 28071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:57.603677 28056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:01:57.603856 28056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "208c6fd301de42439f2814500803a6c7"
format_stamp: "Formatted at 2026-06-20 15:01:57 on dist-test-slave-30t4"
server_key: "859a4cff7a2123eb5063f5873e5e34d3"
server_key_iv: "57a10922070afc36769a65cf5ea68e1e"
server_key_version: "encryptionkey@0"
I20260620 15:01:57.603968 28056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:57.622586 28056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:57.625401 28056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:57.625598 28056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:57.632295 28056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37161
I20260620 15:01:57.632323 28123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37161 every 8 connection(s)
I20260620 15:01:57.633250 28056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:57.635901 28124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:57.637735 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28056
I20260620 15:01:57.637882 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:57.638181 7354 external_mini_cluster.cc:1428] Setting key afb066d5500b09c17a49dfad14741ef9
I20260620 15:01:57.642011 28124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7: Bootstrap starting.
I20260620 15:01:57.644133 28124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:57.644821 28124 log.cc:824] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:57.646515 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:57.639457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42823 (local address 127.0.0.1:37161)
0620 15:01:57.639931 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:57.639947 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:57.639981 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:01:57.640598 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:57.640614 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:57.640933 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:57.641249 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:57.641641 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:57.642635 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:57.643505 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:57.644012 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:57.644179 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:57.644951 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:57.644958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:57.644973 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:57.645358 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:57.645413 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:57.645435 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:57.645541 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:57.645654 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 15:01:57.645851 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":133,"threads_started":1}
I20260620 15:01:57.646651 28124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7: No bootstrap required, opened a new log
I20260620 15:01:57.649158 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:01:57.650612 28124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208c6fd301de42439f2814500803a6c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37161 } }
I20260620 15:01:57.650830 28124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:57.650923 28124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208c6fd301de42439f2814500803a6c7, State: Initialized, Role: FOLLOWER
I20260620 15:01:57.651374 28124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208c6fd301de42439f2814500803a6c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37161 } }
I20260620 15:01:57.651520 28124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:57.651587 28124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:57.651701 28124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:57.652802 28124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208c6fd301de42439f2814500803a6c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37161 } }
I20260620 15:01:57.653112 28124 leader_election.cc:302] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 208c6fd301de42439f2814500803a6c7; no voters:
I20260620 15:01:57.653419 28124 leader_election.cc:288] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:57.653502 28134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:57.653716 28134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [term 1 LEADER]: Becoming Leader. State: Replica: 208c6fd301de42439f2814500803a6c7, State: Running, Role: LEADER
I20260620 15:01:57.654037 28134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208c6fd301de42439f2814500803a6c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37161 } }
I20260620 15:01:57.654630 28124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:57.655282 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:57.650992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39883 (local address 127.0.0.1:37161)
0620 15:01:57.651198 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:57.651204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:57.651218 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:01:57.651362 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:57.651365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:57.651405 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:57.651480 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:57.651733 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:57.652570 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:57.653451 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:57.653784 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:57.653883 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:57.654745 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:57.654749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:57.654755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:57.654769 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:57.654812 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:57.654833 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:57.654905 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:57.655085 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 15:01:57.655155 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":128}
I20260620 15:01:57.656239 28136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 208c6fd301de42439f2814500803a6c7. Latest consensus state: current_term: 1 leader_uuid: "208c6fd301de42439f2814500803a6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208c6fd301de42439f2814500803a6c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37161 } } }
I20260620 15:01:57.656239 28135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "208c6fd301de42439f2814500803a6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208c6fd301de42439f2814500803a6c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37161 } } }
I20260620 15:01:57.656427 28135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:57.656412 28136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:57.656870 28143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:57.659819 28143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:57.664698 28143 catalog_manager.cc:1347] Generated new cluster ID: a2b04ecef68546f09884ae952636cd94
I20260620 15:01:57.664758 28143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:57.672396 28143 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:57.673564 28143 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:57.679239 28143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 208c6fd301de42439f2814500803a6c7: Generated new TSK 0
I20260620 15:01:57.679932 28143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:57.684949 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28056
2026-06-20T15:01:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:57 dist-test-slave-30t4 krb5kdc[28155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:57 dist-test-slave-30t4 krb5kdc[28155](info): set up 2 sockets
Jun 20 15:01:57 dist-test-slave-30t4 krb5kdc[28155](info): commencing operation
krb5kdc: starting...
W20260620 15:01:58.711863 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:01:58 dist-test-slave-30t4 krb5kdc[28155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:01:58.824694 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:58.930225 28168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:58.930509 28168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:58.930588 28168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:58.934552 28168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:58.934636 28168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:58.934675 28168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:58.934727 28168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:58.934767 28168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:58.939779 28168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:01:58.940769 28168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:58.941730 28168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:58.949898 28176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:58.949908 28173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:58.950053 28168 server_base.cc:1034] running on GCE node
W20260620 15:01:58.949972 28174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:58.950649 28168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:58.951684 28168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:58.952844 28168 hybrid_clock.cc:648] HybridClock initialized: now 1781967718952830 us; error 25 us; skew 500 ppm
Jun 20 15:01:58 dist-test-slave-30t4 krb5kdc[28155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:58.955399 28168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:01:58.956347 28168 webserver.cc:460] Webserver started at http://127.0.0.1:32773/ using document root <none> and password file <none>
I20260620 15:01:58.956840 28168 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:58.956897 28168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:58.957048 28168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:58.958963 28168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "79a4048e0c134fcaaf04647f4dcb5ab1"
format_stamp: "Formatted at 2026-06-20 15:01:58 on dist-test-slave-30t4"
server_key: "f3969bec4051250b3ef8c96b2ce0a995"
server_key_iv: "12a4efd6ee6ecee7d96791f28289efc4"
server_key_version: "encryptionkey@0"
I20260620 15:01:58.959502 28168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "79a4048e0c134fcaaf04647f4dcb5ab1"
format_stamp: "Formatted at 2026-06-20 15:01:58 on dist-test-slave-30t4"
server_key: "f3969bec4051250b3ef8c96b2ce0a995"
server_key_iv: "12a4efd6ee6ecee7d96791f28289efc4"
server_key_version: "encryptionkey@0"
I20260620 15:01:58.962931 28168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 15:01:58.965191 28183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:58.966300 28168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:01:58.966393 28168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "79a4048e0c134fcaaf04647f4dcb5ab1"
format_stamp: "Formatted at 2026-06-20 15:01:58 on dist-test-slave-30t4"
server_key: "f3969bec4051250b3ef8c96b2ce0a995"
server_key_iv: "12a4efd6ee6ecee7d96791f28289efc4"
server_key_version: "encryptionkey@0"
I20260620 15:01:58.966470 28168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:58.970705 28168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:58.973356 28168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:58.973488 28168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:58.979907 28168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40889
I20260620 15:01:58.979929 28235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40889 every 8 connection(s)
I20260620 15:01:58.980738 28168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:58.983203 28236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:58.988260 28236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1: Bootstrap starting.
I20260620 15:01:58.989830 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28168
I20260620 15:01:58.989980 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:01:58.990259 7354 external_mini_cluster.cc:1428] Setting key d9bcb1c66a7b0f2114d2e34106ca83bf
I20260620 15:01:58.990610 28236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:58.991482 28236 log.cc:824] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:58.993428 28236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1: No bootstrap required, opened a new log
I20260620 15:01:58.994848 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:58.991237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33869 (local address 127.0.0.1:40889)
0620 15:01:58.991610 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:58.991627 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:58.991667 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:01:58.992462 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:58.992480 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:58.992734 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:58.992992 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:58.993752 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:58.993757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:58.993767 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:58.993777 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:58.993823 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:58.993841 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:58.993921 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:58.994046 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 15:01:58.994198 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
I20260620 15:01:58.997419 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:01:58.997495 28236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4048e0c134fcaaf04647f4dcb5ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } }
I20260620 15:01:58.997730 28236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:58.997790 28236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a4048e0c134fcaaf04647f4dcb5ab1, State: Initialized, Role: FOLLOWER
I20260620 15:01:58.998200 28236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4048e0c134fcaaf04647f4dcb5ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } }
I20260620 15:01:58.998342 28236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:58.998404 28236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:58.998495 28236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:58.999533 28236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4048e0c134fcaaf04647f4dcb5ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } }
I20260620 15:01:58.999889 28236 leader_election.cc:302] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79a4048e0c134fcaaf04647f4dcb5ab1; no voters:
I20260620 15:01:59.000221 28236 leader_election.cc:288] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:59.000274 28246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:59.000494 28246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [term 1 LEADER]: Becoming Leader. State: Replica: 79a4048e0c134fcaaf04647f4dcb5ab1, State: Running, Role: LEADER
I20260620 15:01:59.000797 28246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4048e0c134fcaaf04647f4dcb5ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } }
I20260620 15:01:59.001042 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:58.999379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52867 (local address 127.0.0.1:40889)
0620 15:01:58.999495 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:58.999499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:58.999679 (+ 180us) server_negotiation.cc:407] Connection header received
0620 15:01:58.999839 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:58.999842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:58.999873 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:58.999946 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:59.000649 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:59.000652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:59.000654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:59.000664 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:59.000689 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:59.000700 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:59.000758 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:59.000899 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 15:01:59.000936 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 15:01:59.001398 28236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:59.002492 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79a4048e0c134fcaaf04647f4dcb5ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4048e0c134fcaaf04647f4dcb5ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } } }
I20260620 15:01:59.002668 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:59.002947 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79a4048e0c134fcaaf04647f4dcb5ab1. Latest consensus state: current_term: 1 leader_uuid: "79a4048e0c134fcaaf04647f4dcb5ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4048e0c134fcaaf04647f4dcb5ab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } } }
I20260620 15:01:59.003100 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1 [sys.catalog]: This master's current role is: LEADER
W20260620 15:01:59.005829 28261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:01:59.005910 28261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:01:59.005986 28255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:59.006641 28255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:59.011104 28255 catalog_manager.cc:1347] Generated new cluster ID: cc7ed641e77d4790b48306c9352f3e18
I20260620 15:01:59.011166 28255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:59.023129 28255 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:59.023382 28255 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:59.034789 28255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79a4048e0c134fcaaf04647f4dcb5ab1: Generated new TSK 0
I20260620 15:01:59.035372 28255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:59.044013 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28168
2026-06-20T15:01:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:59 dist-test-slave-30t4 krb5kdc[28267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:59 dist-test-slave-30t4 krb5kdc[28267](info): set up 2 sockets
Jun 20 15:01:59 dist-test-slave-30t4 krb5kdc[28267](info): commencing operation
krb5kdc: starting...
W20260620 15:02:00.084007 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:00 dist-test-slave-30t4 krb5kdc[28267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:02:00.197122 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:00.312245 28280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:00.312472 28280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:00.312541 28280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:00.316433 28280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:00.316484 28280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:00.316504 28280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:00.316519 28280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:00.316552 28280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:00.321473 28280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:00.322402 28280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:00.323331 28280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:00.331310 28285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:00.331387 28286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:00.331368 28280 server_base.cc:1034] running on GCE node
W20260620 15:02:00.331378 28288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:00.332120 28280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:00.333138 28280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:00.334316 28280 hybrid_clock.cc:648] HybridClock initialized: now 1781967720334299 us; error 30 us; skew 500 ppm
Jun 20 15:02:00 dist-test-slave-30t4 krb5kdc[28267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:00.336936 28280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:02:00.337695 28280 webserver.cc:460] Webserver started at http://127.0.0.1:38867/ using document root <none> and password file <none>
I20260620 15:02:00.338202 28280 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:00.338284 28280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:00.338493 28280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:00.340533 28280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "bb2183882bd44ee48a2fbbce96ce99c8"
format_stamp: "Formatted at 2026-06-20 15:02:00 on dist-test-slave-30t4"
server_key: "652ecd4c3dc7f888820db85fa782d937"
server_key_iv: "58ef3de04ad9930fd6ffeee074107da5"
server_key_version: "encryptionkey@0"
I20260620 15:02:00.341055 28280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "bb2183882bd44ee48a2fbbce96ce99c8"
format_stamp: "Formatted at 2026-06-20 15:02:00 on dist-test-slave-30t4"
server_key: "652ecd4c3dc7f888820db85fa782d937"
server_key_iv: "58ef3de04ad9930fd6ffeee074107da5"
server_key_version: "encryptionkey@0"
I20260620 15:02:00.344544 28280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:02:00.346755 28295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:00.347900 28280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:02:00.348021 28280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "bb2183882bd44ee48a2fbbce96ce99c8"
format_stamp: "Formatted at 2026-06-20 15:02:00 on dist-test-slave-30t4"
server_key: "652ecd4c3dc7f888820db85fa782d937"
server_key_iv: "58ef3de04ad9930fd6ffeee074107da5"
server_key_version: "encryptionkey@0"
I20260620 15:02:00.348120 28280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:00.362046 28280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:00.365026 28280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:00.365206 28280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:00.371513 28280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46843
I20260620 15:02:00.371567 28347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46843 every 8 connection(s)
I20260620 15:02:00.372455 28280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:00.374969 28348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:00.380132 28348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8: Bootstrap starting.
I20260620 15:02:00.382385 28348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:00.382689 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28280
I20260620 15:02:00.382843 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:00.383124 7354 external_mini_cluster.cc:1428] Setting key 4f04e76617edd2a2a82792758da8f31d
I20260620 15:02:00.383217 28348 log.cc:824] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:00.385349 28348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8: No bootstrap required, opened a new log
I20260620 15:02:00.389401 28348 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2183882bd44ee48a2fbbce96ce99c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260620 15:02:00.389604 28348 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:00.389659 28348 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2183882bd44ee48a2fbbce96ce99c8, State: Initialized, Role: FOLLOWER
I20260620 15:02:00.390156 28348 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2183882bd44ee48a2fbbce96ce99c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260620 15:02:00.390336 28348 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:00.390441 28348 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:00.390551 28348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:00.391743 28348 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2183882bd44ee48a2fbbce96ce99c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260620 15:02:00.392093 28348 leader_election.cc:302] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb2183882bd44ee48a2fbbce96ce99c8; no voters:
Jun 20 15:02:00 dist-test-slave-30t4 krb5kdc[28267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:02:00.392431 28348 leader_election.cc:288] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:00.392488 28353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:00.392697 28353 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [term 1 LEADER]: Becoming Leader. State: Replica: bb2183882bd44ee48a2fbbce96ce99c8, State: Running, Role: LEADER
I20260620 15:02:00.393028 28353 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2183882bd44ee48a2fbbce96ce99c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260620 15:02:00.393639 28348 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:00.394749 28354 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb2183882bd44ee48a2fbbce96ce99c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2183882bd44ee48a2fbbce96ce99c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } } }
I20260620 15:02:00.394877 28354 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:00.395227 28355 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb2183882bd44ee48a2fbbce96ce99c8. Latest consensus state: current_term: 1 leader_uuid: "bb2183882bd44ee48a2fbbce96ce99c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2183882bd44ee48a2fbbce96ce99c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } } }
I20260620 15:02:00.395377 28355 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:00.395680 28362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:00.398736 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:00.384193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:60977 (local address 127.0.0.1:46843)
0620 15:02:00.384567 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:00.384585 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:00.384864 (+ 279us) server_negotiation.cc:407] Connection header received
0620 15:02:00.385613 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:00.385630 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:00.385948 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:00.386207 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:00.387122 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:00.388197 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:00.389383 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:00.389935 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:00.390063 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:00.393066 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:00.393073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:00.393091 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:00.393123 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:00.395306 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:00.396193 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:00.396199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:00.396204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:00.396355 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:00.396719 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:00.396723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:00.396725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:00.397181 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:00.397418 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:00.397644 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 15:02:00.397852 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260620 15:02:00.399335 28362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:00.401754 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:00.405440 28362 catalog_manager.cc:1347] Generated new cluster ID: 6cd1c7b4177c4b48926bdfe7c9759c96
I20260620 15:02:00.405498 28362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:00.411532 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:00.403945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:44473 (local address 127.0.0.1:46843)
0620 15:02:00.404035 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:00.404039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:00.404275 (+ 236us) server_negotiation.cc:407] Connection header received
0620 15:02:00.404375 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:00.404378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:00.404426 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:00.404507 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:00.405271 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:00.405972 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:00.406747 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:00.406988 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:00.407092 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:00.408470 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:00.408474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:00.408477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:00.408500 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:00.409867 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:00.410315 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:00.410318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:00.410320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:00.410395 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:00.410702 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:00.410705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:00.410707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:00.410867 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:00.411008 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:00.411270 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 15:02:00.411360 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 15:02:00.425434 28362 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:00.426352 28362 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:00.436590 28362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb2183882bd44ee48a2fbbce96ce99c8: Generated new TSK 0
I20260620 15:02:00.437216 28362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:00.444603 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28280
W20260620 15:02:00.450254 28265 connection.cc:535] client connection to 127.0.0.1:46843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46843): Connection reset by peer (error 104)
2026-06-20T15:02:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:00 dist-test-slave-30t4 krb5kdc[28379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:00 dist-test-slave-30t4 krb5kdc[28379](info): set up 2 sockets
Jun 20 15:02:00 dist-test-slave-30t4 krb5kdc[28379](info): commencing operation
krb5kdc: starting...
W20260620 15:02:01.476043 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:01 dist-test-slave-30t4 krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:02:01.591003 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:01.696491 28392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:01.696768 28392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:01.696853 28392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:01.700965 28392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:01.701035 28392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:01.701071 28392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:01.701120 28392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:01.701157 28392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:01.706341 28392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:01.707295 28392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:01.708305 28392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:01.716468 28398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:01.716467 28400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:01.716631 28392 server_base.cc:1034] running on GCE node
W20260620 15:02:01.716491 28397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:01.717285 28392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:01.718313 28392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:01.719472 28392 hybrid_clock.cc:648] HybridClock initialized: now 1781967721719458 us; error 25 us; skew 500 ppm
Jun 20 15:02:01 dist-test-slave-30t4 krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:01.729226 28392 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:02:01.730064 28392 webserver.cc:460] Webserver started at http://127.0.0.1:35667/ using document root <none> and password file <none>
I20260620 15:02:01.730597 28392 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:01.730655 28392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:01.730943 28392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:01.732957 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "d305de13d217400b846ed7eca6ae2ef4"
format_stamp: "Formatted at 2026-06-20 15:02:01 on dist-test-slave-30t4"
server_key: "a64a708ab64ed19284c0fc8558b5bc68"
server_key_iv: "7b6da928e7588e74a164e4499387be90"
server_key_version: "encryptionkey@0"
I20260620 15:02:01.733489 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "d305de13d217400b846ed7eca6ae2ef4"
format_stamp: "Formatted at 2026-06-20 15:02:01 on dist-test-slave-30t4"
server_key: "a64a708ab64ed19284c0fc8558b5bc68"
server_key_iv: "7b6da928e7588e74a164e4499387be90"
server_key_version: "encryptionkey@0"
I20260620 15:02:01.737003 28392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:02:01.739244 28407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:01.740422 28392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:02:01.740550 28392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "d305de13d217400b846ed7eca6ae2ef4"
format_stamp: "Formatted at 2026-06-20 15:02:01 on dist-test-slave-30t4"
server_key: "a64a708ab64ed19284c0fc8558b5bc68"
server_key_iv: "7b6da928e7588e74a164e4499387be90"
server_key_version: "encryptionkey@0"
I20260620 15:02:01.740664 28392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:01.753979 28392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:01.756596 28392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:01.756780 28392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:01.763352 28392 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35845
I20260620 15:02:01.763378 28459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35845 every 8 connection(s)
I20260620 15:02:01.764232 28392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:01.766402 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28392
I20260620 15:02:01.766539 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:01.766826 7354 external_mini_cluster.cc:1428] Setting key 8c605aa09c64fbb8aeead6af729f9642
I20260620 15:02:01.766932 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:01.774294 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4: Bootstrap starting.
I20260620 15:02:01.776928 28460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:01.777209 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:01.768211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:51197 (local address 127.0.0.1:35845)
0620 15:02:01.769008 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:01.769026 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:01.769067 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:02:01.769995 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:01.770019 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:01.770452 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:01.770825 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:01.771216 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:01.772689 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:01.773537 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:01.774206 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:01.774387 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:01.775259 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:01.775276 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:01.775290 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:01.775816 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:01.775878 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:01.775907 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:01.776033 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:01.776181 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:02:01.776312 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":125,"threads_started":1}
I20260620 15:02:01.777743 28460 log.cc:824] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:01.779817 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4: No bootstrap required, opened a new log
I20260620 15:02:01.779982 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:01.783855 28460 raft_consensus.cc:348] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305de13d217400b846ed7eca6ae2ef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35845 } }
I20260620 15:02:01.784077 28460 raft_consensus.cc:374] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:01.784166 28460 raft_consensus.cc:729] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305de13d217400b846ed7eca6ae2ef4, State: Initialized, Role: FOLLOWER
I20260620 15:02:01.784612 28460 consensus_queue.cc:260] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305de13d217400b846ed7eca6ae2ef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35845 } }
I20260620 15:02:01.784781 28460 raft_consensus.cc:388] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:01.784855 28460 raft_consensus.cc:482] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:01.784950 28460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:01.785859 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:01.781858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:50913 (local address 127.0.0.1:35845)
0620 15:02:01.781970 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:01.781974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:01.782086 (+ 112us) server_negotiation.cc:407] Connection header received
0620 15:02:01.782168 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:01.782171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:01.782211 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:01.782300 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:01.782511 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:01.783244 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:01.784204 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:01.784535 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:01.784642 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:01.785464 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:01.785468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:01.785470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:01.785482 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:01.785507 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:01.785519 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:01.785613 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:01.785698 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 15:02:01.785727 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:02:01.785979 28460 raft_consensus.cc:504] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305de13d217400b846ed7eca6ae2ef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35845 } }
I20260620 15:02:01.786302 28460 leader_election.cc:302] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305de13d217400b846ed7eca6ae2ef4; no voters:
I20260620 15:02:01.786580 28460 leader_election.cc:288] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:01.786675 28470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:01.786887 28470 raft_consensus.cc:686] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [term 1 LEADER]: Becoming Leader. State: Replica: d305de13d217400b846ed7eca6ae2ef4, State: Running, Role: LEADER
I20260620 15:02:01.787177 28470 consensus_queue.cc:237] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305de13d217400b846ed7eca6ae2ef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35845 } }
I20260620 15:02:01.787726 28460 sys_catalog.cc:564] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:01.788934 28471 sys_catalog.cc:455] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d305de13d217400b846ed7eca6ae2ef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305de13d217400b846ed7eca6ae2ef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35845 } } }
I20260620 15:02:01.788940 28472 sys_catalog.cc:455] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d305de13d217400b846ed7eca6ae2ef4. Latest consensus state: current_term: 1 leader_uuid: "d305de13d217400b846ed7eca6ae2ef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305de13d217400b846ed7eca6ae2ef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35845 } } }
I20260620 15:02:01.789162 28472 sys_catalog.cc:458] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:01.789202 28471 sys_catalog.cc:458] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:01.789861 28481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:01.792454 28481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:01.797039 28481 catalog_manager.cc:1347] Generated new cluster ID: 356f43634a9f4c85a63e65f464e8e55f
I20260620 15:02:01.797102 28481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:01.811739 28481 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:01.812949 28481 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:01.823431 28481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d305de13d217400b846ed7eca6ae2ef4: Generated new TSK 0
I20260620 15:02:01.824225 28481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:01.836807 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28392
W20260620 15:02:01.842535 28377 connection.cc:535] client connection to 127.0.0.1:35845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35845): Connection reset by peer (error 104)
2026-06-20T15:02:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:01 dist-test-slave-30t4 krb5kdc[28491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:01 dist-test-slave-30t4 krb5kdc[28491](info): set up 2 sockets
Jun 20 15:02:01 dist-test-slave-30t4 krb5kdc[28491](info): commencing operation
krb5kdc: starting...
W20260620 15:02:02.880061 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:02 dist-test-slave-30t4 krb5kdc[28491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:02:02.996739 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:03.104529 28504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:03.104784 28504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:03.104863 28504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:03.108834 28504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:03.108902 28504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:03.108937 28504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:03.108986 28504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:03.109023 28504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:03.114120 28504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:03.115046 28504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:03.116046 28504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:03.123998 28509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:03.124027 28512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:03.124094 28504 server_base.cc:1034] running on GCE node
W20260620 15:02:03.124027 28510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:03.124683 28504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:03.125685 28504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:03.126866 28504 hybrid_clock.cc:648] HybridClock initialized: now 1781967723126839 us; error 37 us; skew 500 ppm
Jun 20 15:02:03 dist-test-slave-30t4 krb5kdc[28491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:03.129449 28504 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:02:03.130183 28504 webserver.cc:460] Webserver started at http://127.0.0.1:39279/ using document root <none> and password file <none>
I20260620 15:02:03.130693 28504 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:03.130752 28504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:03.130903 28504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:03.132835 28504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "06a43042bda54deea7c638ecaae36c8f"
format_stamp: "Formatted at 2026-06-20 15:02:03 on dist-test-slave-30t4"
server_key: "f5686c2423720ba80a720dd9edc4ab7d"
server_key_iv: "903480009548df2fb8d09eeae7364db3"
server_key_version: "encryptionkey@0"
I20260620 15:02:03.133315 28504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "06a43042bda54deea7c638ecaae36c8f"
format_stamp: "Formatted at 2026-06-20 15:02:03 on dist-test-slave-30t4"
server_key: "f5686c2423720ba80a720dd9edc4ab7d"
server_key_iv: "903480009548df2fb8d09eeae7364db3"
server_key_version: "encryptionkey@0"
I20260620 15:02:03.136721 28504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:02:03.138891 28519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:03.140067 28504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:02:03.140159 28504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "06a43042bda54deea7c638ecaae36c8f"
format_stamp: "Formatted at 2026-06-20 15:02:03 on dist-test-slave-30t4"
server_key: "f5686c2423720ba80a720dd9edc4ab7d"
server_key_iv: "903480009548df2fb8d09eeae7364db3"
server_key_version: "encryptionkey@0"
I20260620 15:02:03.140234 28504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:03.144374 28504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:03.146929 28504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:03.147061 28504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:03.154011 28504 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39419
I20260620 15:02:03.154036 28571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39419 every 8 connection(s)
I20260620 15:02:03.154848 28504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:03.157403 28572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:03.162034 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28504
I20260620 15:02:03.162175 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:03.162469 7354 external_mini_cluster.cc:1428] Setting key df42460e09582182205827f3c7ee8157
I20260620 15:02:03.162720 28572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f: Bootstrap starting.
I20260620 15:02:03.165077 28572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:03.165827 28572 log.cc:824] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:03.167326 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:03.163704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:33027 (local address 127.0.0.1:39419)
0620 15:02:03.164200 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:03.164214 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:03.164248 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:02:03.164852 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:03.164869 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:03.165158 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:03.165493 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:03.166242 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:03.166248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:03.166266 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:03.166276 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:03.166324 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:03.166342 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:03.166447 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:03.166553 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 15:02:03.166714 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":131,"threads_started":1}
I20260620 15:02:03.167698 28572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f: No bootstrap required, opened a new log
I20260620 15:02:03.170199 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:03.171640 28572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a43042bda54deea7c638ecaae36c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } }
I20260620 15:02:03.171911 28572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:03.171983 28572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a43042bda54deea7c638ecaae36c8f, State: Initialized, Role: FOLLOWER
I20260620 15:02:03.172453 28572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a43042bda54deea7c638ecaae36c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } }
I20260620 15:02:03.172621 28572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:03.172691 28572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:03.172794 28572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:03.173823 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:03.172102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:37347 (local address 127.0.0.1:39419)
0620 15:02:03.172204 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:03.172208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:03.172452 (+ 244us) server_negotiation.cc:407] Connection header received
0620 15:02:03.172560 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:03.172562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:03.172589 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:03.172679 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:03.173440 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:03.173443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:03.173444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:03.173454 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:03.173477 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:03.173486 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:03.173547 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:03.173660 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 15:02:03.173690 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 15:02:03.173992 28572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a43042bda54deea7c638ecaae36c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } }
I20260620 15:02:03.174311 28572 leader_election.cc:302] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a43042bda54deea7c638ecaae36c8f; no voters:
I20260620 15:02:03.174615 28572 leader_election.cc:288] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:03.174705 28582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:03.174947 28582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [term 1 LEADER]: Becoming Leader. State: Replica: 06a43042bda54deea7c638ecaae36c8f, State: Running, Role: LEADER
I20260620 15:02:03.175256 28582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a43042bda54deea7c638ecaae36c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } }
I20260620 15:02:03.175756 28572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:03.176899 28584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a43042bda54deea7c638ecaae36c8f. Latest consensus state: current_term: 1 leader_uuid: "06a43042bda54deea7c638ecaae36c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a43042bda54deea7c638ecaae36c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } } }
I20260620 15:02:03.176899 28583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a43042bda54deea7c638ecaae36c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a43042bda54deea7c638ecaae36c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } } }
I20260620 15:02:03.177042 28583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:03.177042 28584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:03.177419 28591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:03.179988 28591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:03.185150 28591 catalog_manager.cc:1347] Generated new cluster ID: d8225d9ee1ce4559afefa5d68a12a7a7
I20260620 15:02:03.185209 28591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:03.207346 28591 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:03.207659 28591 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:03.216755 28591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a43042bda54deea7c638ecaae36c8f: Generated new TSK 0
I20260620 15:02:03.217398 28591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:03.226003 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28504
2026-06-20T15:02:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:03 dist-test-slave-30t4 krb5kdc[28603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:03 dist-test-slave-30t4 krb5kdc[28603](info): set up 2 sockets
Jun 20 15:02:03 dist-test-slave-30t4 krb5kdc[28603](info): commencing operation
krb5kdc: starting...
W20260620 15:02:04.259992 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:04 dist-test-slave-30t4 krb5kdc[28603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:02:04.371941 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:04.476564 28616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:04.476802 28616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:04.476841 28616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:04.480696 28616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:04.480746 28616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:04.480764 28616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:04.480780 28616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:04.480795 28616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:04.485774 28616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:04.486728 28616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:04.487689 28616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:02:04.496011 28616 server_base.cc:1034] running on GCE node
W20260620 15:02:04.495987 28622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:04.496130 28621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:04.496003 28624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:04.496758 28616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:04.497802 28616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:04.498997 28616 hybrid_clock.cc:648] HybridClock initialized: now 1781967724498980 us; error 27 us; skew 500 ppm
Jun 20 15:02:04 dist-test-slave-30t4 krb5kdc[28603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:04.501709 28616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:02:04.502497 28616 webserver.cc:460] Webserver started at http://127.0.0.1:36357/ using document root <none> and password file <none>
I20260620 15:02:04.502976 28616 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:04.503031 28616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:04.503183 28616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:04.505139 28616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "f7725e71ad6e45c98422eaeee4dfee89"
format_stamp: "Formatted at 2026-06-20 15:02:04 on dist-test-slave-30t4"
server_key: "e208efadc354832d10b65d79f57ece50"
server_key_iv: "c02a04fe5a8e8ee7141f1d9ba96a1823"
server_key_version: "encryptionkey@0"
I20260620 15:02:04.505636 28616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "f7725e71ad6e45c98422eaeee4dfee89"
format_stamp: "Formatted at 2026-06-20 15:02:04 on dist-test-slave-30t4"
server_key: "e208efadc354832d10b65d79f57ece50"
server_key_iv: "c02a04fe5a8e8ee7141f1d9ba96a1823"
server_key_version: "encryptionkey@0"
I20260620 15:02:04.509068 28616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:02:04.511307 28631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:04.512495 28616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:02:04.512602 28616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "f7725e71ad6e45c98422eaeee4dfee89"
format_stamp: "Formatted at 2026-06-20 15:02:04 on dist-test-slave-30t4"
server_key: "e208efadc354832d10b65d79f57ece50"
server_key_iv: "c02a04fe5a8e8ee7141f1d9ba96a1823"
server_key_version: "encryptionkey@0"
I20260620 15:02:04.512686 28616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:04.533824 28616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:04.536886 28616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:04.537027 28616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:04.543540 28616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37043
I20260620 15:02:04.543565 28683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37043 every 8 connection(s)
I20260620 15:02:04.544469 28616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:04.547056 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28616
I20260620 15:02:04.547199 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:04.547222 28684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:04.547531 7354 external_mini_cluster.cc:1428] Setting key c822c587e97ea9073a9c7753df54e47a
I20260620 15:02:04.554239 28684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89: Bootstrap starting.
Jun 20 15:02:04 dist-test-slave-30t4 krb5kdc[28603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:02:04.556717 28684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:04.557557 28684 log.cc:824] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:04.559415 28684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89: No bootstrap required, opened a new log
I20260620 15:02:04.561709 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:04.548789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:39699 (local address 127.0.0.1:37043)
0620 15:02:04.549181 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:04.549200 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:04.549239 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:02:04.549957 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:04.549979 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:04.550391 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:04.550738 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:04.551364 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:04.552859 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:04.553662 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:04.554315 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:04.554504 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:04.557180 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:04.557191 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:04.557203 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:04.557228 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:04.558877 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:04.559401 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:04.559406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:04.559410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:04.559578 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:04.559921 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:04.559923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:04.559925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:04.560275 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:04.560421 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:04.560773 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 15:02:04.561036 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260620 15:02:04.563242 28684 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7725e71ad6e45c98422eaeee4dfee89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260620 15:02:04.563508 28684 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:04.563570 28684 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7725e71ad6e45c98422eaeee4dfee89, State: Initialized, Role: FOLLOWER
I20260620 15:02:04.564052 28684 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7725e71ad6e45c98422eaeee4dfee89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260620 15:02:04.564221 28684 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:04.564292 28684 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:04.564392 28684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:04.564565 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:04.565246 28684 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7725e71ad6e45c98422eaeee4dfee89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260620 15:02:04.565584 28684 leader_election.cc:302] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7725e71ad6e45c98422eaeee4dfee89; no voters:
I20260620 15:02:04.565899 28684 leader_election.cc:288] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:04.565953 28693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:04.566155 28693 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [term 1 LEADER]: Becoming Leader. State: Replica: f7725e71ad6e45c98422eaeee4dfee89, State: Running, Role: LEADER
I20260620 15:02:04.566901 28693 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7725e71ad6e45c98422eaeee4dfee89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260620 15:02:04.567134 28684 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:04.568490 28694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7725e71ad6e45c98422eaeee4dfee89. Latest consensus state: current_term: 1 leader_uuid: "f7725e71ad6e45c98422eaeee4dfee89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7725e71ad6e45c98422eaeee4dfee89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } } }
I20260620 15:02:04.568619 28694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:04.568994 28695 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7725e71ad6e45c98422eaeee4dfee89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7725e71ad6e45c98422eaeee4dfee89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } } }
I20260620 15:02:04.569460 28695 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89 [sys.catalog]: This master's current role is: LEADER
W20260620 15:02:04.572124 28709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:04.572199 28709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:04.572321 28706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:04.573212 28706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:04.576363 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:04.566557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:56457 (local address 127.0.0.1:37043)
0620 15:02:04.566640 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:04.566645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:04.568069 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 15:02:04.568275 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:04.568280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:04.568341 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:04.568401 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:04.569512 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:04.570125 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:04.570903 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:04.571144 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:04.571262 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:04.572818 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:04.572823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:04.572826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:04.572850 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:04.574563 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:04.575080 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:04.575084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:04.575086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:04.575173 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:04.575496 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:04.575499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:04.575501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:04.575669 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:04.575850 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:04.576111 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:02:04.576214 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 15:02:04.578415 28706 catalog_manager.cc:1347] Generated new cluster ID: c3d38ebbfd9c4baca68f937935743b88
I20260620 15:02:04.578472 28706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:04.598783 28706 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:04.599920 28706 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:04.604765 28706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7725e71ad6e45c98422eaeee4dfee89: Generated new TSK 0
I20260620 15:02:04.605391 28706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:04.612637 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28616
W20260620 15:02:04.617915 28601 connection.cc:535] client connection to 127.0.0.1:37043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37043): Connection reset by peer (error 104)
2026-06-20T15:02:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:04 dist-test-slave-30t4 krb5kdc[28715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:04 dist-test-slave-30t4 krb5kdc[28715](info): set up 2 sockets
Jun 20 15:02:04 dist-test-slave-30t4 krb5kdc[28715](info): commencing operation
krb5kdc: starting...
W20260620 15:02:05.647995 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:05 dist-test-slave-30t4 krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:02:05.760597 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:05.865235 28728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:05.865470 28728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:05.865510 28728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:05.869336 28728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:05.869380 28728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:05.869400 28728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:05.869414 28728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:05.869429 28728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:05.874339 28728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:05.875205 28728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:05.876120 28728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:05.884488 28734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:05.884536 28733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:05.884485 28736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:05.884761 28728 server_base.cc:1034] running on GCE node
I20260620 15:02:05.885274 28728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:05.886250 28728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:05.887406 28728 hybrid_clock.cc:648] HybridClock initialized: now 1781967725887375 us; error 42 us; skew 500 ppm
Jun 20 15:02:05 dist-test-slave-30t4 krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:05.890087 28728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:02:05.890970 28728 webserver.cc:460] Webserver started at http://127.0.0.1:34937/ using document root <none> and password file <none>
I20260620 15:02:05.891505 28728 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:05.891589 28728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:05.891861 28728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:05.893834 28728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "96b95bd35a9a44eda3c56c688b88a783"
format_stamp: "Formatted at 2026-06-20 15:02:05 on dist-test-slave-30t4"
server_key: "b63cf7491817c2c4a7286dc2f953d34c"
server_key_iv: "c9983e99f54cd7705555577cda4cc196"
server_key_version: "encryptionkey@0"
I20260620 15:02:05.894389 28728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "96b95bd35a9a44eda3c56c688b88a783"
format_stamp: "Formatted at 2026-06-20 15:02:05 on dist-test-slave-30t4"
server_key: "b63cf7491817c2c4a7286dc2f953d34c"
server_key_iv: "c9983e99f54cd7705555577cda4cc196"
server_key_version: "encryptionkey@0"
I20260620 15:02:05.897907 28728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:02:05.900213 28743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:05.901443 28728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:02:05.901580 28728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "96b95bd35a9a44eda3c56c688b88a783"
format_stamp: "Formatted at 2026-06-20 15:02:05 on dist-test-slave-30t4"
server_key: "b63cf7491817c2c4a7286dc2f953d34c"
server_key_iv: "c9983e99f54cd7705555577cda4cc196"
server_key_version: "encryptionkey@0"
I20260620 15:02:05.901693 28728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:05.929584 28728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:05.932902 28728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:05.933112 28728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:05.940156 28728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35925
I20260620 15:02:05.940179 28795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35925 every 8 connection(s)
I20260620 15:02:05.941062 28728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:05.943709 28796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:05.946203 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28728
I20260620 15:02:05.946344 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:05.946651 7354 external_mini_cluster.cc:1428] Setting key 9c16dd63323de8ee8d0247e8d379f966
I20260620 15:02:05.949858 28796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783: Bootstrap starting.
I20260620 15:02:05.952396 28796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:05.953202 28796 log.cc:824] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:05.955323 28796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783: No bootstrap required, opened a new log
I20260620 15:02:05.956522 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:05.948013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:34831 (local address 127.0.0.1:35925)
0620 15:02:05.948412 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:05.948430 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:05.948490 (+ 60us) server_negotiation.cc:407] Connection header received
0620 15:02:05.949275 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:05.949296 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:05.949700 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:05.950050 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:05.950449 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:05.951930 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:05.952870 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:05.953575 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:05.953692 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:05.954624 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:05.954631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:05.954645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:05.955106 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:05.955183 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:05.955212 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:05.955343 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:05.955503 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 15:02:05.955634 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260620 15:02:05.959331 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:05.959450 28796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b95bd35a9a44eda3c56c688b88a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } }
I20260620 15:02:05.959671 28796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:05.959741 28796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b95bd35a9a44eda3c56c688b88a783, State: Initialized, Role: FOLLOWER
I20260620 15:02:05.960178 28796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b95bd35a9a44eda3c56c688b88a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } }
I20260620 15:02:05.960292 28796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:05.960373 28796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:05.960480 28796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:05.961314 28796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b95bd35a9a44eda3c56c688b88a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } }
I20260620 15:02:05.961673 28796 leader_election.cc:302] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96b95bd35a9a44eda3c56c688b88a783; no voters:
I20260620 15:02:05.961993 28796 leader_election.cc:288] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:05.962046 28806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:05.962286 28806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [term 1 LEADER]: Becoming Leader. State: Replica: 96b95bd35a9a44eda3c56c688b88a783, State: Running, Role: LEADER
I20260620 15:02:05.962620 28806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b95bd35a9a44eda3c56c688b88a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } }
I20260620 15:02:05.963205 28796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:05.963665 28806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96b95bd35a9a44eda3c56c688b88a783. Latest consensus state: current_term: 1 leader_uuid: "96b95bd35a9a44eda3c56c688b88a783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b95bd35a9a44eda3c56c688b88a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } } }
I20260620 15:02:05.963867 28806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:05.964125 28807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96b95bd35a9a44eda3c56c688b88a783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b95bd35a9a44eda3c56c688b88a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } } }
I20260620 15:02:05.964228 28807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:05.964380 28813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:05.966485 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:05.961438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:44741 (local address 127.0.0.1:35925)
0620 15:02:05.961535 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:05.961539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:05.961764 (+ 225us) server_negotiation.cc:407] Connection header received
0620 15:02:05.961909 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:05.961912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:05.961966 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:05.962046 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:05.962448 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:05.963168 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:05.964390 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:05.964733 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:05.964818 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:05.965961 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:05.965964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:05.965967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:05.965980 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:05.966005 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:05.966020 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:05.966076 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:05.966327 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 15:02:05.966363 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:02:05.967360 28813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:05.972770 28813 catalog_manager.cc:1347] Generated new cluster ID: 3517dda159244e11b63b9b7f67ac59a7
I20260620 15:02:05.972834 28813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:06.005515 28813 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:06.006875 28813 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:06.018054 28813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96b95bd35a9a44eda3c56c688b88a783: Generated new TSK 0
I20260620 15:02:06.018864 28813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:06.024446 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28728
W20260620 15:02:06.030067 28713 connection.cc:535] client connection to 127.0.0.1:35925 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35925): Connection reset by peer (error 104)
2026-06-20T15:02:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:06 dist-test-slave-30t4 krb5kdc[28827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:06 dist-test-slave-30t4 krb5kdc[28827](info): set up 2 sockets
Jun 20 15:02:06 dist-test-slave-30t4 krb5kdc[28827](info): commencing operation
krb5kdc: starting...
W20260620 15:02:07.060194 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:07 dist-test-slave-30t4 krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:02:07.178184 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:07.282326 28840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:07.282608 28840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:07.282682 28840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:07.286631 28840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:07.286732 28840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:07.286772 28840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:07.286830 28840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:07.286886 28840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:07.292045 28840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:07.293032 28840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:07.294015 28840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:07.302271 28848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:07.302327 28840 server_base.cc:1034] running on GCE node
W20260620 15:02:07.302276 28845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:07.302284 28846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:07.303089 28840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:07.304174 28840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:07.305359 28840 hybrid_clock.cc:648] HybridClock initialized: now 1781967727305331 us; error 47 us; skew 500 ppm
Jun 20 15:02:07 dist-test-slave-30t4 krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:07.317860 28840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:02:07.318780 28840 webserver.cc:460] Webserver started at http://127.0.0.1:39861/ using document root <none> and password file <none>
I20260620 15:02:07.319356 28840 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:07.319422 28840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:07.319669 28840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:07.321743 28840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06"
format_stamp: "Formatted at 2026-06-20 15:02:07 on dist-test-slave-30t4"
server_key: "3401e5218c665f9767b2f09ce4d77ef7"
server_key_iv: "07ba9db2773e67fbb85fc9e5b8fe073f"
server_key_version: "encryptionkey@0"
I20260620 15:02:07.322306 28840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06"
format_stamp: "Formatted at 2026-06-20 15:02:07 on dist-test-slave-30t4"
server_key: "3401e5218c665f9767b2f09ce4d77ef7"
server_key_iv: "07ba9db2773e67fbb85fc9e5b8fe073f"
server_key_version: "encryptionkey@0"
I20260620 15:02:07.326092 28840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:02:07.328477 28855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:07.329631 28840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:02:07.329766 28840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06"
format_stamp: "Formatted at 2026-06-20 15:02:07 on dist-test-slave-30t4"
server_key: "3401e5218c665f9767b2f09ce4d77ef7"
server_key_iv: "07ba9db2773e67fbb85fc9e5b8fe073f"
server_key_version: "encryptionkey@0"
I20260620 15:02:07.329871 28840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:07.334337 28840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:07.337067 28840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:07.337248 28840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:07.344161 28840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35515
I20260620 15:02:07.344192 28907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35515 every 8 connection(s)
I20260620 15:02:07.345012 28840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:07.347594 28908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:07.352882 28908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06: Bootstrap starting.
I20260620 15:02:07.353482 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28840
I20260620 15:02:07.353624 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:07.353863 7354 external_mini_cluster.cc:1428] Setting key 1e2bcf0ba64c75bd4d98dab6cefd54dd
I20260620 15:02:07.355502 28908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:07.356280 28908 log.cc:824] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:07.358407 28908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06: No bootstrap required, opened a new log
I20260620 15:02:07.358881 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:07.355136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:34733 (local address 127.0.0.1:35515)
0620 15:02:07.355549 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:07.355563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:07.355596 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:02:07.356251 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:07.356268 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:07.356531 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:07.356798 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:07.357566 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:07.357572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:07.357586 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:07.357597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:07.357658 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:07.357688 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:07.357794 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:07.357881 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 15:02:07.358022 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260620 15:02:07.361523 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:07.362854 28908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } }
I20260620 15:02:07.363056 28908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:07.363129 28908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1c96fcc6da4a4abd422ca3cbb44d06, State: Initialized, Role: FOLLOWER
I20260620 15:02:07.363628 28908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } }
I20260620 15:02:07.363827 28908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:07.363900 28908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:07.363996 28908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:07.364902 28908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } }
I20260620 15:02:07.365131 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:07.363469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:45239 (local address 127.0.0.1:35515)
0620 15:02:07.363588 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:07.363592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:07.363752 (+ 160us) server_negotiation.cc:407] Connection header received
0620 15:02:07.363933 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:07.363938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:07.363973 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:07.364059 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:07.364793 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:07.364799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:07.364804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:07.364817 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:07.364845 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:07.364860 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:07.364957 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:07.364996 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 15:02:07.365030 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 15:02:07.365263 28908 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c1c96fcc6da4a4abd422ca3cbb44d06; no voters:
I20260620 15:02:07.365568 28908 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:07.365660 28918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:07.365862 28918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [term 1 LEADER]: Becoming Leader. State: Replica: 9c1c96fcc6da4a4abd422ca3cbb44d06, State: Running, Role: LEADER
I20260620 15:02:07.366178 28918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } }
I20260620 15:02:07.366746 28908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:07.367209 28918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1c96fcc6da4a4abd422ca3cbb44d06. Latest consensus state: current_term: 1 leader_uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } } }
I20260620 15:02:07.367359 28918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:07.367733 28924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:07.367760 28919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1c96fcc6da4a4abd422ca3cbb44d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35515 } } }
I20260620 15:02:07.367956 28919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:07.370770 28924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:07.376183 28924 catalog_manager.cc:1347] Generated new cluster ID: 005012bb1d7e4d2f9330b0fe68983ddc
I20260620 15:02:07.376271 28924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:07.394227 28924 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:07.394573 28924 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:07.400290 28924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c1c96fcc6da4a4abd422ca3cbb44d06: Generated new TSK 0
I20260620 15:02:07.400923 28924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:07.412756 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28840
2026-06-20T15:02:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:07 dist-test-slave-30t4 krb5kdc[28939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:07 dist-test-slave-30t4 krb5kdc[28939](info): set up 2 sockets
Jun 20 15:02:07 dist-test-slave-30t4 krb5kdc[28939](info): commencing operation
krb5kdc: starting...
W20260620 15:02:08.452204 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:08 dist-test-slave-30t4 krb5kdc[28939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:02:08.570470 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:45599
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:36161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:08.677590 28952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:08.677809 28952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:08.677846 28952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:08.681830 28952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:08.681880 28952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:08.681900 28952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:08.681921 28952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:08.681972 28952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:08.686980 28952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:36161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:45599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:45599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:08.687939 28952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:08.688862 28952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:08.696915 28957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:08.696913 28960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:08.697009 28952 server_base.cc:1034] running on GCE node
W20260620 15:02:08.696961 28958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:08.697649 28952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:08.698644 28952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:08.699857 28952 hybrid_clock.cc:648] HybridClock initialized: now 1781967728699838 us; error 24 us; skew 500 ppm
Jun 20 15:02:08 dist-test-slave-30t4 krb5kdc[28939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967728, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:08.702456 28952 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:02:08.703269 28952 webserver.cc:460] Webserver started at http://127.7.46.190:45737/ using document root <none> and password file <none>
I20260620 15:02:08.703902 28952 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:08.703992 28952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:08.704212 28952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:08.706140 28952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "a58f704be22b437d87dc01c450b79bf3"
format_stamp: "Formatted at 2026-06-20 15:02:08 on dist-test-slave-30t4"
server_key: "bab6e7cd97badadfc9b6c2671cc08b04"
server_key_iv: "173ff968b530b710e43b93d49174f420"
server_key_version: "encryptionkey@0"
I20260620 15:02:08.706707 28952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "a58f704be22b437d87dc01c450b79bf3"
format_stamp: "Formatted at 2026-06-20 15:02:08 on dist-test-slave-30t4"
server_key: "bab6e7cd97badadfc9b6c2671cc08b04"
server_key_iv: "173ff968b530b710e43b93d49174f420"
server_key_version: "encryptionkey@0"
I20260620 15:02:08.710227 28952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 15:02:08.712530 28967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:08.713749 28952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:02:08.713903 28952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "a58f704be22b437d87dc01c450b79bf3"
format_stamp: "Formatted at 2026-06-20 15:02:08 on dist-test-slave-30t4"
server_key: "bab6e7cd97badadfc9b6c2671cc08b04"
server_key_iv: "173ff968b530b710e43b93d49174f420"
server_key_version: "encryptionkey@0"
I20260620 15:02:08.714012 28952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:08.726646 28952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:08.729355 28952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:08.729559 28952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:08.736348 28952 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:45599
I20260620 15:02:08.736375 29019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:45599 every 8 connection(s)
I20260620 15:02:08.737233 28952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:08.739864 29020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:08.745038 29020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3: Bootstrap starting.
I20260620 15:02:08.746084 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 28952
I20260620 15:02:08.746230 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:08.746610 7354 external_mini_cluster.cc:1428] Setting key 909ccde7bd90f0f5e39ce84d36eaa12e
I20260620 15:02:08.748642 29020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:08.749702 29020 log.cc:824] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:08.752115 29020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3: No bootstrap required, opened a new log
I20260620 15:02:08.756856 29020 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58f704be22b437d87dc01c450b79bf3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45599 } }
I20260620 15:02:08.757124 29020 raft_consensus.cc:374] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:08.757197 29020 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58f704be22b437d87dc01c450b79bf3, State: Initialized, Role: FOLLOWER
Jun 20 15:02:08 dist-test-slave-30t4 krb5kdc[28939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:02:08.757687 29020 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58f704be22b437d87dc01c450b79bf3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45599 } }
I20260620 15:02:08.757856 29020 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:08.757939 29020 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:08.758054 29020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:08.758909 29020 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58f704be22b437d87dc01c450b79bf3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45599 } }
I20260620 15:02:08.759240 29020 leader_election.cc:302] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58f704be22b437d87dc01c450b79bf3; no voters:
I20260620 15:02:08.759577 29020 leader_election.cc:288] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:08.759625 29025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:08.759810 29025 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [term 1 LEADER]: Becoming Leader. State: Replica: a58f704be22b437d87dc01c450b79bf3, State: Running, Role: LEADER
I20260620 15:02:08.760164 29025 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58f704be22b437d87dc01c450b79bf3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45599 } }
I20260620 15:02:08.760818 29020 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:08.761920 29027 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a58f704be22b437d87dc01c450b79bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58f704be22b437d87dc01c450b79bf3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45599 } } }
I20260620 15:02:08.761926 29026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58f704be22b437d87dc01c450b79bf3. Latest consensus state: current_term: 1 leader_uuid: "a58f704be22b437d87dc01c450b79bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58f704be22b437d87dc01c450b79bf3" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45599 } } }
I20260620 15:02:08.762126 29026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:08.762411 29027 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:08.762746 29034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:08.763960 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:08.747843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47927 (local address 127.7.46.190:45599)
0620 15:02:08.748449 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:08.748469 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:08.748516 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:02:08.749540 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:08.749567 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:08.750019 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:08.750380 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:08.751318 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:08.752708 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:08.753966 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:08.754655 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:08.754813 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:08.758062 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:08.758068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:08.758083 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:08.758137 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:08.760068 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:08.760618 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:08.760623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:08.760629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:08.760762 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:08.761100 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:08.761103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:08.761104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:08.761652 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:08.761880 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:08.762802 (+ 922us) server_negotiation.cc:299] Negotiation successful
0620 15:02:08.763015 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":127,"threads_started":1}
I20260620 15:02:08.766091 29034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:08.766659 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:08.771422 29034 catalog_manager.cc:1347] Generated new cluster ID: 2320ea64e6f24d059e9f6d84465b10c7
I20260620 15:02:08.771497 29034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:08.784176 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:08.775898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47519 (local address 127.7.46.190:45599)
0620 15:02:08.776004 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:08.776009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:08.776022 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:02:08.776058 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:08.776060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:08.776100 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:08.776181 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:08.776938 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:08.777736 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:08.778799 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:08.779163 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:08.779261 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:08.780841 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:08.780845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:08.780848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:08.780872 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:08.782432 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:08.782945 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:08.782949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:08.782951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:08.783031 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:08.783338 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:08.783341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:08.783343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:08.783535 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:08.783681 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:08.783931 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 15:02:08.784020 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:02:08.786801 29034 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:08.787721 29034 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:08.797751 29034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58f704be22b437d87dc01c450b79bf3: Generated new TSK 0
I20260620 15:02:08.798480 29034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:08.804023 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 28952
W20260620 15:02:08.809511 28937 connection.cc:535] client connection to 127.7.46.190:45599 recv error: Network error: failed to read from TLS socket (remote: 127.7.46.190:45599): Connection reset by peer (error 104)
2026-06-20T15:02:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:08 dist-test-slave-30t4 krb5kdc[29051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:08 dist-test-slave-30t4 krb5kdc[29051](info): set up 2 sockets
Jun 20 15:02:08 dist-test-slave-30t4 krb5kdc[29051](info): commencing operation
krb5kdc: starting...
W20260620 15:02:09.831980 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:09 dist-test-slave-30t4 krb5kdc[29051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:02:09.944967 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:41161
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:41161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:10.049794 29064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:10.050073 29064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:10.050158 29064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:10.054100 29064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:10.054184 29064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:10.054229 29064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:10.054261 29064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:10.054291 29064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:10.059244 29064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:41161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:10.060199 29064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:10.061410 29064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:02:10.069953 29064 server_base.cc:1034] running on GCE node
W20260620 15:02:10.069886 29069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:10.069998 29072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:10.069886 29070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:10.070750 29064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:10.071758 29064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:10.072989 29064 hybrid_clock.cc:648] HybridClock initialized: now 1781967730072961 us; error 40 us; skew 500 ppm
Jun 20 15:02:10 dist-test-slave-30t4 krb5kdc[29051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967730, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:10.075511 29064 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:02:10.076383 29064 webserver.cc:460] Webserver started at http://127.7.46.190:38667/ using document root <none> and password file <none>
I20260620 15:02:10.076912 29064 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:10.077001 29064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:10.077224 29064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:10.079111 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "bc0c9b447f494f93a554ac1d06ffe6ab"
format_stamp: "Formatted at 2026-06-20 15:02:10 on dist-test-slave-30t4"
server_key: "eca2a2333312ba49216cc396f639bd2a"
server_key_iv: "9eaba6ef7724fd7a9362a79d6e47c17a"
server_key_version: "encryptionkey@0"
I20260620 15:02:10.079674 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "bc0c9b447f494f93a554ac1d06ffe6ab"
format_stamp: "Formatted at 2026-06-20 15:02:10 on dist-test-slave-30t4"
server_key: "eca2a2333312ba49216cc396f639bd2a"
server_key_iv: "9eaba6ef7724fd7a9362a79d6e47c17a"
server_key_version: "encryptionkey@0"
I20260620 15:02:10.083125 29064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:02:10.085393 29079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:10.086553 29064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 15:02:10.086705 29064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "bc0c9b447f494f93a554ac1d06ffe6ab"
format_stamp: "Formatted at 2026-06-20 15:02:10 on dist-test-slave-30t4"
server_key: "eca2a2333312ba49216cc396f639bd2a"
server_key_iv: "9eaba6ef7724fd7a9362a79d6e47c17a"
server_key_version: "encryptionkey@0"
I20260620 15:02:10.086814 29064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:10.103000 29064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:10.106454 29064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:10.106660 29064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:10.113355 29064 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:41161
I20260620 15:02:10.113379 29131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:41161 every 8 connection(s)
I20260620 15:02:10.114197 29064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:10.116714 29132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:10.120157 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 29064
I20260620 15:02:10.120302 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:10.120627 7354 external_mini_cluster.cc:1428] Setting key c6888819193890630b46e9bcdc139700
I20260620 15:02:10.122418 29132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab: Bootstrap starting.
I20260620 15:02:10.124624 29132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:10.125378 29132 log.cc:824] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:10.127045 29132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab: No bootstrap required, opened a new log
I20260620 15:02:10.129408 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:10.121963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33191 (local address 127.7.46.190:41161)
0620 15:02:10.122371 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:10.122386 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:10.122419 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:02:10.123108 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:10.123125 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:10.123440 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:10.123777 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:10.124246 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:10.125423 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:10.126325 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:10.126802 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:10.126977 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:10.127868 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:10.127873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:10.127885 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:10.128233 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:10.128307 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:10.128329 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:10.128464 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:10.128646 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 15:02:10.128776 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260620 15:02:10.130607 29132 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0c9b447f494f93a554ac1d06ffe6ab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41161 } }
I20260620 15:02:10.130856 29132 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:10.130903 29132 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0c9b447f494f93a554ac1d06ffe6ab, State: Initialized, Role: FOLLOWER
I20260620 15:02:10.131367 29132 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0c9b447f494f93a554ac1d06ffe6ab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41161 } }
I20260620 15:02:10.131518 29132 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:10.131601 29132 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:10.131695 29132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:10.132238 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:10.132747 29132 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0c9b447f494f93a554ac1d06ffe6ab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41161 } }
I20260620 15:02:10.133096 29132 leader_election.cc:302] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc0c9b447f494f93a554ac1d06ffe6ab; no voters:
I20260620 15:02:10.133443 29132 leader_election.cc:288] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:10.133553 29137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:10.133800 29137 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [term 1 LEADER]: Becoming Leader. State: Replica: bc0c9b447f494f93a554ac1d06ffe6ab, State: Running, Role: LEADER
I20260620 15:02:10.134156 29137 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0c9b447f494f93a554ac1d06ffe6ab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41161 } }
I20260620 15:02:10.134747 29132 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:10.135389 29137 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc0c9b447f494f93a554ac1d06ffe6ab. Latest consensus state: current_term: 1 leader_uuid: "bc0c9b447f494f93a554ac1d06ffe6ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0c9b447f494f93a554ac1d06ffe6ab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41161 } } }
I20260620 15:02:10.135690 29137 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:10.135829 29142 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc0c9b447f494f93a554ac1d06ffe6ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0c9b447f494f93a554ac1d06ffe6ab" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 41161 } } }
I20260620 15:02:10.135980 29142 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:10.136149 29151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:10.138947 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:10.134336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39469 (local address 127.7.46.190:41161)
0620 15:02:10.134428 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:10.134432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:10.134664 (+ 232us) server_negotiation.cc:407] Connection header received
0620 15:02:10.134765 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:10.134767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:10.134809 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:10.134905 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:10.135282 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:10.136175 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:10.137184 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:10.137484 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:10.137586 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:10.138433 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:10.138436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:10.138439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:10.138452 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:10.138479 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:10.138493 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:10.138577 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:10.138794 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 15:02:10.138826 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":35}
I20260620 15:02:10.139266 29151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:10.148097 29151 catalog_manager.cc:1347] Generated new cluster ID: 25abc7e32dad4af7a2d787e59b3f1df3
I20260620 15:02:10.148173 29151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:10.155810 29151 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:10.156672 29151 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:10.160904 29151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc0c9b447f494f93a554ac1d06ffe6ab: Generated new TSK 0
I20260620 15:02:10.161507 29151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:10.166333 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 29064
W20260620 15:02:10.171622 29049 connection.cc:535] client connection to 127.7.46.190:41161 recv error: Network error: failed to read from TLS socket (remote: 127.7.46.190:41161): Connection reset by peer (error 104)
2026-06-20T15:02:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:10 dist-test-slave-30t4 krb5kdc[29163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:10 dist-test-slave-30t4 krb5kdc[29163](info): set up 2 sockets
Jun 20 15:02:10 dist-test-slave-30t4 krb5kdc[29163](info): commencing operation
krb5kdc: starting...
W20260620 15:02:11.211964 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:11 dist-test-slave-30t4 krb5kdc[29163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:02:11.326440 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:38305
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:11.431273 29176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:11.431532 29176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:11.431609 29176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:11.435580 29176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:11.435668 29176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:11.435714 29176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:11.435750 29176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:11.435798 29176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:11.440845 29176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:38305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:11.441776 29176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:11.442723 29176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:11.450924 29181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:11.450971 29176 server_base.cc:1034] running on GCE node
W20260620 15:02:11.450956 29182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:11.450929 29184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:11.451730 29176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:11.452768 29176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:11.453933 29176 hybrid_clock.cc:648] HybridClock initialized: now 1781967731453919 us; error 24 us; skew 500 ppm
Jun 20 15:02:11 dist-test-slave-30t4 krb5kdc[29163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967731, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:11.456612 29176 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:02:11.457410 29176 webserver.cc:460] Webserver started at http://127.7.46.190:35301/ using document root <none> and password file <none>
I20260620 15:02:11.457937 29176 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:11.458024 29176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:11.458246 29176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:11.460196 29176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "08879523e8f246179fd656aa3c017d6a"
format_stamp: "Formatted at 2026-06-20 15:02:11 on dist-test-slave-30t4"
server_key: "418773f29c9967dc175ab6bdcef2b55e"
server_key_iv: "3693b4f6525d831f126181507bbbda3c"
server_key_version: "encryptionkey@0"
I20260620 15:02:11.460731 29176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "08879523e8f246179fd656aa3c017d6a"
format_stamp: "Formatted at 2026-06-20 15:02:11 on dist-test-slave-30t4"
server_key: "418773f29c9967dc175ab6bdcef2b55e"
server_key_iv: "3693b4f6525d831f126181507bbbda3c"
server_key_version: "encryptionkey@0"
I20260620 15:02:11.464187 29176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:02:11.466387 29191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:11.467496 29176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:02:11.467645 29176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "08879523e8f246179fd656aa3c017d6a"
format_stamp: "Formatted at 2026-06-20 15:02:11 on dist-test-slave-30t4"
server_key: "418773f29c9967dc175ab6bdcef2b55e"
server_key_iv: "3693b4f6525d831f126181507bbbda3c"
server_key_version: "encryptionkey@0"
I20260620 15:02:11.467751 29176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:11.472118 29176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:11.474749 29176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:11.474934 29176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:11.481389 29176 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:38305
I20260620 15:02:11.481498 29243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:38305 every 8 connection(s)
I20260620 15:02:11.482251 29176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:11.484766 29244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:11.489810 29244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a: Bootstrap starting.
I20260620 15:02:11.491642 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 29176
I20260620 15:02:11.491806 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:11.492116 7354 external_mini_cluster.cc:1428] Setting key 6bad59d8b6b34df63d709c97e4d89f74
I20260620 15:02:11.492194 29244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:11.493026 29244 log.cc:824] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:11.495128 29244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a: No bootstrap required, opened a new log
I20260620 15:02:11.497197 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:11.493222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55595 (local address 127.7.46.190:38305)
0620 15:02:11.493630 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:11.493645 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:11.493676 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:02:11.494431 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:11.494454 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:11.494789 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:11.495120 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:11.495880 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:11.495886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:11.495900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:11.495912 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:11.495975 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:11.496003 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:11.496108 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:11.496210 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 15:02:11.496347 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260620 15:02:11.499408 29244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08879523e8f246179fd656aa3c017d6a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38305 } }
I20260620 15:02:11.499650 29244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:11.499740 29244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08879523e8f246179fd656aa3c017d6a, State: Initialized, Role: FOLLOWER
I20260620 15:02:11.499938 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:11.500192 29244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08879523e8f246179fd656aa3c017d6a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38305 } }
I20260620 15:02:11.500336 29244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:11.500419 29244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:11.500514 29244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:11.501384 29244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08879523e8f246179fd656aa3c017d6a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38305 } }
I20260620 15:02:11.501696 29244 leader_election.cc:302] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08879523e8f246179fd656aa3c017d6a; no voters:
I20260620 15:02:11.501997 29244 leader_election.cc:288] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:11.502168 29253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:11.502414 29253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [term 1 LEADER]: Becoming Leader. State: Replica: 08879523e8f246179fd656aa3c017d6a, State: Running, Role: LEADER
I20260620 15:02:11.502738 29253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08879523e8f246179fd656aa3c017d6a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38305 } }
I20260620 15:02:11.503352 29244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:11.503882 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:11.501937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51097 (local address 127.7.46.190:38305)
0620 15:02:11.502060 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:11.502064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:11.502277 (+ 213us) server_negotiation.cc:407] Connection header received
0620 15:02:11.502508 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:11.502511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:11.502542 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:11.502587 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:11.503393 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:11.503396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:11.503399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:11.503413 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:11.503439 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:11.503465 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:11.503523 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:11.503721 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 15:02:11.503759 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 15:02:11.503937 29253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08879523e8f246179fd656aa3c017d6a. Latest consensus state: current_term: 1 leader_uuid: "08879523e8f246179fd656aa3c017d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08879523e8f246179fd656aa3c017d6a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38305 } } }
I20260620 15:02:11.504065 29253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:11.504351 29255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08879523e8f246179fd656aa3c017d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08879523e8f246179fd656aa3c017d6a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 38305 } } }
I20260620 15:02:11.504438 29255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:11.505589 29261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:11.508526 29261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:11.513334 29261 catalog_manager.cc:1347] Generated new cluster ID: 02450328db5546bea0d73e201f861e03
I20260620 15:02:11.513404 29261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:11.534626 29261 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:11.534931 29261 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:11.548772 29261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08879523e8f246179fd656aa3c017d6a: Generated new TSK 0
I20260620 15:02:11.549471 29261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:11.560060 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 29176
2026-06-20T15:02:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:11 dist-test-slave-30t4 krb5kdc[29275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:11 dist-test-slave-30t4 krb5kdc[29275](info): set up 2 sockets
Jun 20 15:02:11 dist-test-slave-30t4 krb5kdc[29275](info): commencing operation
krb5kdc: starting...
W20260620 15:02:12.588106 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:12 dist-test-slave-30t4 krb5kdc[29275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:02:12.707144 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34643
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:34643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:12.815459 29288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:12.815696 29288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:12.815737 29288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:12.819614 29288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:12.819665 29288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:12.819684 29288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:12.819700 29288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:12.819753 29288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:12.824769 29288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:34643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:12.825681 29288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:12.826603 29288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:12.834872 29293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:12.834849 29296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:12.834861 29294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:12.835238 29288 server_base.cc:1034] running on GCE node
I20260620 15:02:12.835739 29288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:12.836815 29288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:12.837998 29288 hybrid_clock.cc:648] HybridClock initialized: now 1781967732837986 us; error 33 us; skew 500 ppm
Jun 20 15:02:12 dist-test-slave-30t4 krb5kdc[29275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:12.845211 29288 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:02:12.846084 29288 webserver.cc:460] Webserver started at http://127.7.46.190:34515/ using document root <none> and password file <none>
I20260620 15:02:12.846623 29288 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:12.846711 29288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:12.846926 29288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:12.848886 29288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "ebd52b12b9cd42bc8678732a3c7930fd"
format_stamp: "Formatted at 2026-06-20 15:02:12 on dist-test-slave-30t4"
server_key: "1539a62f4218ee9166f7285baddff5ba"
server_key_iv: "4fa64c676f7e3dd522d57cff767ce0fa"
server_key_version: "encryptionkey@0"
I20260620 15:02:12.849432 29288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "ebd52b12b9cd42bc8678732a3c7930fd"
format_stamp: "Formatted at 2026-06-20 15:02:12 on dist-test-slave-30t4"
server_key: "1539a62f4218ee9166f7285baddff5ba"
server_key_iv: "4fa64c676f7e3dd522d57cff767ce0fa"
server_key_version: "encryptionkey@0"
I20260620 15:02:12.852874 29288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 15:02:12.855050 29303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:12.856201 29288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:02:12.856345 29288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "ebd52b12b9cd42bc8678732a3c7930fd"
format_stamp: "Formatted at 2026-06-20 15:02:12 on dist-test-slave-30t4"
server_key: "1539a62f4218ee9166f7285baddff5ba"
server_key_iv: "4fa64c676f7e3dd522d57cff767ce0fa"
server_key_version: "encryptionkey@0"
I20260620 15:02:12.856446 29288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:12.881464 29288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:12.884238 29288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:12.884425 29288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:12.891180 29288 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:34643
I20260620 15:02:12.891194 29355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:34643 every 8 connection(s)
I20260620 15:02:12.892095 29288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:12.892537 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 29288
I20260620 15:02:12.892688 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:12.892926 7354 external_mini_cluster.cc:1428] Setting key 3f138c056832c4bb4cdd027187f5df90
I20260620 15:02:12.895128 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:12.901300 29356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd: Bootstrap starting.
Jun 20 15:02:12 dist-test-slave-30t4 krb5kdc[29275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:02:12.903697 29356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:12.904397 29356 log.cc:824] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:12.906468 29356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd: No bootstrap required, opened a new log
I20260620 15:02:12.907126 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:12.894162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43617 (local address 127.7.46.190:34643)
0620 15:02:12.894554 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:12.894583 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:12.894623 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:02:12.895444 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:12.895465 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:12.895901 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:12.896251 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:12.896900 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:12.898373 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:12.899238 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:12.899953 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:12.900117 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:12.902716 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:12.902720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:12.902731 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:12.902756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:12.904408 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:12.904904 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:12.904912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:12.904921 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:12.905054 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:12.905353 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:12.905356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:12.905357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:12.905709 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:12.905835 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:12.906142 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 15:02:12.906418 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260620 15:02:12.909973 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:12.910497 29356 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd52b12b9cd42bc8678732a3c7930fd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34643 } }
I20260620 15:02:12.910697 29356 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:12.910779 29356 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd52b12b9cd42bc8678732a3c7930fd, State: Initialized, Role: FOLLOWER
I20260620 15:02:12.911238 29356 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd52b12b9cd42bc8678732a3c7930fd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34643 } }
I20260620 15:02:12.911391 29356 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:12.911454 29356 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:12.911545 29356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:12.912585 29356 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd52b12b9cd42bc8678732a3c7930fd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34643 } }
I20260620 15:02:12.912911 29356 leader_election.cc:302] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebd52b12b9cd42bc8678732a3c7930fd; no voters:
I20260620 15:02:12.913201 29356 leader_election.cc:288] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:12.913322 29366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:12.913607 29366 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [term 1 LEADER]: Becoming Leader. State: Replica: ebd52b12b9cd42bc8678732a3c7930fd, State: Running, Role: LEADER
I20260620 15:02:12.913935 29366 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd52b12b9cd42bc8678732a3c7930fd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34643 } }
I20260620 15:02:12.914440 29356 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:12.915047 29366 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd52b12b9cd42bc8678732a3c7930fd. Latest consensus state: current_term: 1 leader_uuid: "ebd52b12b9cd42bc8678732a3c7930fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd52b12b9cd42bc8678732a3c7930fd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34643 } } }
I20260620 15:02:12.915210 29366 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:12.916226 29368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebd52b12b9cd42bc8678732a3c7930fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd52b12b9cd42bc8678732a3c7930fd" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34643 } } }
I20260620 15:02:12.916393 29368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd [sys.catalog]: This master's current role is: LEADER
W20260620 15:02:12.919471 29381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:12.919548 29381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:12.919677 29375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:12.920408 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:12.912011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41781 (local address 127.7.46.190:34643)
0620 15:02:12.912120 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:12.912124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:12.912345 (+ 221us) server_negotiation.cc:407] Connection header received
0620 15:02:12.912458 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:12.912461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:12.912500 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:12.912586 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:12.913299 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:12.914044 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:12.915146 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:12.915546 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:12.915672 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:12.917287 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:12.917292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:12.917295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:12.917320 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:12.918752 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:12.919157 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:12.919164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:12.919167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:12.919248 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:12.919509 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:12.919514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:12.919515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:12.919748 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:12.919910 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:12.920141 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:02:12.920252 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:02:12.920606 29375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:12.926136 29375 catalog_manager.cc:1347] Generated new cluster ID: 0cafbf4a2d634ff0be74b09f5f9a387a
I20260620 15:02:12.926199 29375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:12.938184 29375 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:12.939077 29375 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:12.945487 29375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebd52b12b9cd42bc8678732a3c7930fd: Generated new TSK 0
I20260620 15:02:12.946131 29375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:12.955127 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 29288
2026-06-20T15:02:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:13 dist-test-slave-30t4 krb5kdc[29387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:13 dist-test-slave-30t4 krb5kdc[29387](info): set up 2 sockets
Jun 20 15:02:13 dist-test-slave-30t4 krb5kdc[29387](info): commencing operation
krb5kdc: starting...
W20260620 15:02:13.988032 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:14 dist-test-slave-30t4 krb5kdc[29387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:02:14.104025 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:36811
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:36811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:14.209143 29400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:14.209393 29400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:14.209461 29400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:14.213393 29400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:14.213474 29400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:14.213512 29400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:14.213567 29400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:14.213604 29400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:14.218572 29400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:36811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:36811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:14.219512 29400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:14.220502 29400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:14.228567 29405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:14.228535 29408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:14.228618 29406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:14.228703 29400 server_base.cc:1034] running on GCE node
I20260620 15:02:14.229260 29400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:14.230250 29400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:14.231403 29400 hybrid_clock.cc:648] HybridClock initialized: now 1781967734231385 us; error 30 us; skew 500 ppm
Jun 20 15:02:14 dist-test-slave-30t4 krb5kdc[29387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967734, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:14.234050 29400 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:02:14.234830 29400 webserver.cc:460] Webserver started at http://127.7.46.190:42347/ using document root <none> and password file <none>
I20260620 15:02:14.235301 29400 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:14.235354 29400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:14.235498 29400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:14.237361 29400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "5ce17e79e83a4a24a5fe12aea78666be"
format_stamp: "Formatted at 2026-06-20 15:02:14 on dist-test-slave-30t4"
server_key: "6d80040bd0638ae1d9fed1652d968ddf"
server_key_iv: "10d22f5dcc63285b8d06eb4acc309d0d"
server_key_version: "encryptionkey@0"
I20260620 15:02:14.237823 29400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "5ce17e79e83a4a24a5fe12aea78666be"
format_stamp: "Formatted at 2026-06-20 15:02:14 on dist-test-slave-30t4"
server_key: "6d80040bd0638ae1d9fed1652d968ddf"
server_key_iv: "10d22f5dcc63285b8d06eb4acc309d0d"
server_key_version: "encryptionkey@0"
I20260620 15:02:14.241149 29400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:02:14.243319 29415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:14.244539 29400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:02:14.244671 29400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "5ce17e79e83a4a24a5fe12aea78666be"
format_stamp: "Formatted at 2026-06-20 15:02:14 on dist-test-slave-30t4"
server_key: "6d80040bd0638ae1d9fed1652d968ddf"
server_key_iv: "10d22f5dcc63285b8d06eb4acc309d0d"
server_key_version: "encryptionkey@0"
I20260620 15:02:14.244776 29400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:14.270736 29400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:14.273350 29400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:14.273540 29400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:14.280282 29400 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:36811
I20260620 15:02:14.280303 29467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:36811 every 8 connection(s)
I20260620 15:02:14.281123 29400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:14.283620 29468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:14.288690 29468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be: Bootstrap starting.
I20260620 15:02:14.289393 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 29400
I20260620 15:02:14.289521 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:14.289844 7354 external_mini_cluster.cc:1428] Setting key 47aa2e21fa49a0cbf3d4fb4f07bca7f5
I20260620 15:02:14.291463 29468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:14.292232 29468 log.cc:824] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:14.293965 29468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be: No bootstrap required, opened a new log
I20260620 15:02:14.297509 29468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce17e79e83a4a24a5fe12aea78666be" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36811 } }
I20260620 15:02:14.297765 29468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:14.297832 29468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce17e79e83a4a24a5fe12aea78666be, State: Initialized, Role: FOLLOWER
I20260620 15:02:14.298305 29468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce17e79e83a4a24a5fe12aea78666be" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36811 } }
I20260620 15:02:14.298421 29468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:14.298499 29468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:14.298576 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.290974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36767 (local address 127.7.46.190:36811)
0620 15:02:14.291368 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.291382 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.291415 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:02:14.292088 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.292105 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.292478 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.292780 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:14.293181 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.294229 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.295405 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.295931 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.296096 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.296903 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:14.296908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:14.296928 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:14.297271 (+ 343us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:14.297317 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:14.297339 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:14.297444 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.297564 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.297788 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260620 15:02:14.298612 29468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:14.299372 29468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce17e79e83a4a24a5fe12aea78666be" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36811 } }
I20260620 15:02:14.299695 29468 leader_election.cc:302] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ce17e79e83a4a24a5fe12aea78666be; no voters:
I20260620 15:02:14.300017 29468 leader_election.cc:288] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:14.300087 29473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:14.300271 29473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [term 1 LEADER]: Becoming Leader. State: Replica: 5ce17e79e83a4a24a5fe12aea78666be, State: Running, Role: LEADER
I20260620 15:02:14.300544 29473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce17e79e83a4a24a5fe12aea78666be" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36811 } }
I20260620 15:02:14.301079 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:14.301177 29468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:14.301621 29473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ce17e79e83a4a24a5fe12aea78666be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce17e79e83a4a24a5fe12aea78666be" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36811 } } }
I20260620 15:02:14.301775 29473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:14.302049 29473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce17e79e83a4a24a5fe12aea78666be. Latest consensus state: current_term: 1 leader_uuid: "5ce17e79e83a4a24a5fe12aea78666be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce17e79e83a4a24a5fe12aea78666be" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 36811 } } }
I20260620 15:02:14.302215 29473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be [sys.catalog]: This master's current role is: LEADER
W20260620 15:02:14.305986 29493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:14.306074 29493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:14.306181 29479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:14.307076 29479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:14.307750 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.304023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44201 (local address 127.7.46.190:36811)
0620 15:02:14.304122 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.304126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.304297 (+ 171us) server_negotiation.cc:407] Connection header received
0620 15:02:14.304408 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.304411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.304447 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.304537 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:14.304773 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.305254 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.306019 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.306340 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.306442 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.307340 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:14.307344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:14.307347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:14.307360 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:14.307385 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:14.307398 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:14.307474 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.307587 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.307641 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 15:02:14.312198 29479 catalog_manager.cc:1347] Generated new cluster ID: dda066e8837f4bf793788761cea1afb2
I20260620 15:02:14.312255 29479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:14.323433 29479 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:14.324586 29479 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:14.329882 29479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ce17e79e83a4a24a5fe12aea78666be: Generated new TSK 0
I20260620 15:02:14.330544 29479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:14.337541 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 29400
2026-06-20T15:02:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:14 dist-test-slave-30t4 krb5kdc[29499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:14 dist-test-slave-30t4 krb5kdc[29499](info): set up 2 sockets
Jun 20 15:02:14 dist-test-slave-30t4 krb5kdc[29499](info): commencing operation
krb5kdc: starting...
W20260620 15:02:15.372043 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:15 dist-test-slave-30t4 krb5kdc[29499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:02:15.490404 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37703
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:15.595471 29512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:15.595758 29512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:15.595855 29512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:15.599865 29512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:15.599951 29512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:15.600003 29512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:15.600061 29512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:15.600101 29512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:15.605206 29512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:37703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:37703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:15.606189 29512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:15.607163 29512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:15.615252 29520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:15.615360 29518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:15.615300 29517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:15.615442 29512 server_base.cc:1034] running on GCE node
I20260620 15:02:15.616201 29512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:15.617166 29512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:15.618322 29512 hybrid_clock.cc:648] HybridClock initialized: now 1781967735618309 us; error 25 us; skew 500 ppm
Jun 20 15:02:15 dist-test-slave-30t4 krb5kdc[29499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967735, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:15.621062 29512 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:02:15.621899 29512 webserver.cc:460] Webserver started at http://127.7.46.190:42999/ using document root <none> and password file <none>
I20260620 15:02:15.622419 29512 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:15.622478 29512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:15.622709 29512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:15.624699 29512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "d327667d064d47b1b8835d93cb00e82a"
format_stamp: "Formatted at 2026-06-20 15:02:15 on dist-test-slave-30t4"
server_key: "b085327422357bbbd1992c9d18996d33"
server_key_iv: "4a1bf85e2eb043ca010e59ddced51e4a"
server_key_version: "encryptionkey@0"
I20260620 15:02:15.625257 29512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "d327667d064d47b1b8835d93cb00e82a"
format_stamp: "Formatted at 2026-06-20 15:02:15 on dist-test-slave-30t4"
server_key: "b085327422357bbbd1992c9d18996d33"
server_key_iv: "4a1bf85e2eb043ca010e59ddced51e4a"
server_key_version: "encryptionkey@0"
I20260620 15:02:15.628793 29512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:02:15.631047 29527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:15.632357 29512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:02:15.632491 29512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "d327667d064d47b1b8835d93cb00e82a"
format_stamp: "Formatted at 2026-06-20 15:02:15 on dist-test-slave-30t4"
server_key: "b085327422357bbbd1992c9d18996d33"
server_key_iv: "4a1bf85e2eb043ca010e59ddced51e4a"
server_key_version: "encryptionkey@0"
I20260620 15:02:15.632603 29512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:15.636993 29512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:15.640105 29512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:15.640288 29512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:15.647174 29512 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:37703
I20260620 15:02:15.647198 29579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:37703 every 8 connection(s)
I20260620 15:02:15.648069 29512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:15.650570 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:15.655673 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 29512
I20260620 15:02:15.655766 29580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a: Bootstrap starting.
I20260620 15:02:15.655831 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:15.656126 7354 external_mini_cluster.cc:1428] Setting key 9aaf185e081f5191fbb306b732b34719
I20260620 15:02:15.658226 29580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:15.658900 29580 log.cc:824] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:15.661005 29580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a: No bootstrap required, opened a new log
I20260620 15:02:15.661199 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:15.657380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38047 (local address 127.7.46.190:37703)
0620 15:02:15.657790 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:15.657803 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:15.657836 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:02:15.658509 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:15.658527 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:15.658786 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:15.659094 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:15.659907 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:15.659913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:15.659926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:15.659937 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:15.659996 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:15.660022 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:15.660125 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:15.660214 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 15:02:15.660360 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":132,"threads_started":1}
I20260620 15:02:15.663708 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:15.664713 29580 raft_consensus.cc:348] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d327667d064d47b1b8835d93cb00e82a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37703 } }
I20260620 15:02:15.664969 29580 raft_consensus.cc:374] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:15.665056 29580 raft_consensus.cc:729] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d327667d064d47b1b8835d93cb00e82a, State: Initialized, Role: FOLLOWER
I20260620 15:02:15.665514 29580 consensus_queue.cc:260] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d327667d064d47b1b8835d93cb00e82a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37703 } }
I20260620 15:02:15.665702 29580 raft_consensus.cc:388] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:15.665777 29580 raft_consensus.cc:482] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:15.665894 29580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:15.666893 29580 raft_consensus.cc:504] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d327667d064d47b1b8835d93cb00e82a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37703 } }
I20260620 15:02:15.667207 29580 leader_election.cc:302] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d327667d064d47b1b8835d93cb00e82a; no voters:
I20260620 15:02:15.667505 29580 leader_election.cc:288] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:15.667570 29590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:15.667718 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:15.665937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34067 (local address 127.7.46.190:37703)
0620 15:02:15.666072 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:15.666078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:15.666233 (+ 155us) server_negotiation.cc:407] Connection header received
0620 15:02:15.666313 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:15.666316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:15.666346 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:15.666420 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:15.667126 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:15.667129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:15.667132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:02:15.667143 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:15.667168 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:02:15.667179 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:15.667271 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:15.667555 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:02:15.667595 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:02:15.667757 29590 raft_consensus.cc:686] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [term 1 LEADER]: Becoming Leader. State: Replica: d327667d064d47b1b8835d93cb00e82a, State: Running, Role: LEADER
I20260620 15:02:15.668337 29590 consensus_queue.cc:237] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d327667d064d47b1b8835d93cb00e82a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37703 } }
I20260620 15:02:15.668743 29580 sys_catalog.cc:564] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:15.669974 29592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d327667d064d47b1b8835d93cb00e82a. Latest consensus state: current_term: 1 leader_uuid: "d327667d064d47b1b8835d93cb00e82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d327667d064d47b1b8835d93cb00e82a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37703 } } }
I20260620 15:02:15.670101 29592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:15.670467 29591 sys_catalog.cc:455] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d327667d064d47b1b8835d93cb00e82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d327667d064d47b1b8835d93cb00e82a" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 37703 } } }
I20260620 15:02:15.670559 29591 sys_catalog.cc:458] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a [sys.catalog]: This master's current role is: LEADER
W20260620 15:02:15.673034 29605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:15.673118 29605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:15.673187 29600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:15.673799 29600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:15.678781 29600 catalog_manager.cc:1347] Generated new cluster ID: ad12f8a80613408eb0499a025d0f67e3
I20260620 15:02:15.678853 29600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:15.694697 29600 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:15.695003 29600 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:15.699918 29600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d327667d064d47b1b8835d93cb00e82a: Generated new TSK 0
I20260620 15:02:15.700719 29600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:02:15.704062 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 29512
2026-06-20T15:02:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1366 ms)
[----------] 18 tests from AuthTokenIssuingTest (24948 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:15 dist-test-slave-30t4 krb5kdc[29611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:15 dist-test-slave-30t4 krb5kdc[29611](info): set up 2 sockets
Jun 20 15:02:15 dist-test-slave-30t4 krb5kdc[29611](info): commencing operation
krb5kdc: starting...
W20260620 15:02:16.744079 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:16 dist-test-slave-30t4 krb5kdc[29611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:02:16.863296 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:45527
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:45527,127.7.46.189:41567,127.7.46.188:46241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:16.968958 29624 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:02:16.969267 29624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:16.969340 29624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:16.969420 29624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:16.973366 29624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:16.973448 29624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:16.973491 29624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:16.973544 29624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:16.973574 29624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:16.978528 29624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:45527,127.7.46.189:41567,127.7.46.188:46241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:16.979481 29624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:16.980450 29624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:16.988343 29632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:16.988368 29629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:16.988369 29630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:16.988674 29624 server_base.cc:1034] running on GCE node
I20260620 15:02:16.989183 29624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:16.990252 29624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:16.991456 29624 hybrid_clock.cc:648] HybridClock initialized: now 1781967736991445 us; error 28 us; skew 500 ppm
Jun 20 15:02:16 dist-test-slave-30t4 krb5kdc[29611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967736, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:16.994228 29624 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:02:16.995087 29624 webserver.cc:460] Webserver started at http://127.7.46.190:41193/ using document root <none> and password file <none>
I20260620 15:02:16.995625 29624 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:16.995711 29624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:16.995956 29624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:16.997875 29624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "003c166f8822498aa53ff631ca29af61"
format_stamp: "Formatted at 2026-06-20 15:02:16 on dist-test-slave-30t4"
server_key: "2a9bb89c8378323e9ccd3d2113ed1b0a"
server_key_iv: "b5b08ba10f677ea5a8962761a55fdd87"
server_key_version: "encryptionkey@0"
I20260620 15:02:16.998440 29624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "003c166f8822498aa53ff631ca29af61"
format_stamp: "Formatted at 2026-06-20 15:02:16 on dist-test-slave-30t4"
server_key: "2a9bb89c8378323e9ccd3d2113ed1b0a"
server_key_iv: "b5b08ba10f677ea5a8962761a55fdd87"
server_key_version: "encryptionkey@0"
I20260620 15:02:17.002010 29624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 15:02:17.004371 29639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:17.005541 29624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:02:17.005694 29624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "003c166f8822498aa53ff631ca29af61"
format_stamp: "Formatted at 2026-06-20 15:02:16 on dist-test-slave-30t4"
server_key: "2a9bb89c8378323e9ccd3d2113ed1b0a"
server_key_iv: "b5b08ba10f677ea5a8962761a55fdd87"
server_key_version: "encryptionkey@0"
I20260620 15:02:17.005800 29624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:17.016974 29624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:17.020120 29624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:17.020306 29624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:17.027060 29624 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:45527
I20260620 15:02:17.027341 29691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:45527 every 8 connection(s)
I20260620 15:02:17.027976 29624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:17.028925 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 29624
I20260620 15:02:17.029066 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:17.029387 7354 external_mini_cluster.cc:1428] Setting key 00b192b6a9521814b6e7170b39c73120
I20260620 15:02:17.031250 29692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:17.035043 29692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.46.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.189@KRBTEST.COM" created.
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967736, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:02:17.050871 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.038190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.7.46.190:45527)
0620 15:02:17.038440 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.038449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.038482 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:02:17.039313 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.039318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.039486 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.039729 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.040852 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.041723 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.042709 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.043345 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.043535 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.045998 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.046005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.046017 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.046046 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.048080 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.048544 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.048549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.048550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.048648 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.048985 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.048989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.048991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.049349 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.049449 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.049906 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.050204 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260620 15:02:17.050869 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.037441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45527 (local address 127.0.0.1:51966)
0620 15:02:17.037944 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.037987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.038963 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.039721 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.039726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.040223 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.040660 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.040683 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.041863 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.041867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.042587 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.042597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.043536 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.043540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.043686 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.044421 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.044437 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.045838 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.048182 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.048193 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.048200 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.048416 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.048753 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.048756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.048757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.048870 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.049461 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.049466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.049649 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.049788 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.050047 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":45,"thread_start_us":121,"threads_started":1}
I20260620 15:02:17.053332 29692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:17.053978 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.053768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55464)
0620 15:02:17.053867 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.053929 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 15:02:17.054179 29640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
I20260620 15:02:17.055006 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.054768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55476)
0620 15:02:17.054881 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.054933 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:02:17.055132 29692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:41567: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab.
Entry for principal kudu/127.7.46.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab.
I20260620 15:02:17.056262 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.189:41567
--webserver_interface=127.7.46.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab
--principal=kudu/127.7.46.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:45527,127.7.46.189:41567,127.7.46.188:46241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:02:17.104384 29692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } attempt: 1
I20260620 15:02:17.105193 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.104905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55480)
0620 15:02:17.105048 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.105117 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:02:17.105882 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.105676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55482)
0620 15:02:17.105781 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.105832 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 15:02:17.106009 29692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:41567: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
W20260620 15:02:17.161777 29699 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:02:17.162060 29699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:17.162134 29699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:17.162220 29699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:17.166159 29699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:17.166237 29699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:17.166273 29699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:17.166309 29699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:17.166339 29699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:17.171289 29699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:45527,127.7.46.189:41567,127.7.46.188:46241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab
--principal=kudu/127.7.46.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.189:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.46.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:17.172308 29699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:17.173238 29699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:02:17.174163 29692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } attempt: 2
I20260620 15:02:17.174930 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.174652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55486)
0620 15:02:17.174800 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.174861 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:02:17.175590 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.175384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55500)
0620 15:02:17.175493 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.175541 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 15:02:17.175698 29692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:41567: Network error: Client connection negotiation failed: client connection to 127.7.46.189:41567: connect: Connection refused (error 111)
W20260620 15:02:17.181376 29707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:17.181417 29704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:17.181427 29705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:17.181581 29699 server_base.cc:1034] running on GCE node
I20260620 15:02:17.182252 29699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:17.183238 29699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:17.184428 29699 hybrid_clock.cc:648] HybridClock initialized: now 1781967737184415 us; error 22 us; skew 500 ppm
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:17.187083 29699 init.cc:377] Logged in from keytab as kudu/127.7.46.189@KRBTEST.COM (short username kudu)
I20260620 15:02:17.187893 29699 webserver.cc:460] Webserver started at http://127.7.46.189:42237/ using document root <none> and password file <none>
I20260620 15:02:17.188370 29699 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:17.188423 29699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:17.188571 29699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:17.190380 29699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/data/instance:
uuid: "e889215feed14e6cb390ae8d84e5a299"
format_stamp: "Formatted at 2026-06-20 15:02:17 on dist-test-slave-30t4"
server_key: "5a5058df2646991b10289d7bfedbe269"
server_key_iv: "d6bab66e2095cbd4b36f40a5ecdea4bc"
server_key_version: "encryptionkey@0"
I20260620 15:02:17.190883 29699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/wal/instance:
uuid: "e889215feed14e6cb390ae8d84e5a299"
format_stamp: "Formatted at 2026-06-20 15:02:17 on dist-test-slave-30t4"
server_key: "5a5058df2646991b10289d7bfedbe269"
server_key_iv: "d6bab66e2095cbd4b36f40a5ecdea4bc"
server_key_version: "encryptionkey@0"
I20260620 15:02:17.194427 29699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 15:02:17.196734 29714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:17.197969 29699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 15:02:17.198112 29699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/wal
uuid: "e889215feed14e6cb390ae8d84e5a299"
format_stamp: "Formatted at 2026-06-20 15:02:17 on dist-test-slave-30t4"
server_key: "5a5058df2646991b10289d7bfedbe269"
server_key_iv: "d6bab66e2095cbd4b36f40a5ecdea4bc"
server_key_version: "encryptionkey@0"
I20260620 15:02:17.198210 29699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:17.209661 29699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:17.212690 29699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:17.212886 29699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:17.219534 29699 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.189:41567
I20260620 15:02:17.219859 29766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.189:41567 every 8 connection(s)
I20260620 15:02:17.220502 29699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
I20260620 15:02:17.221390 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 29699
I20260620 15:02:17.221534 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-1/wal/instance
I20260620 15:02:17.221803 7354 external_mini_cluster.cc:1428] Setting key 707a72f50c6cb3313a02b751d4f1c843
I20260620 15:02:17.223278 29767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:17.226752 29767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.46.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.188@KRBTEST.COM" created.
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:02:17.241146 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.228571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.7.46.190:45527)
0620 15:02:17.228701 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.228705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.229685 (+ 980us) server_negotiation.cc:407] Connection header received
0620 15:02:17.230712 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.230715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.230780 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.230878 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.232207 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.232765 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.234429 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.234669 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.234781 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.237501 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.237504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.237506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.237528 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.238931 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.239577 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.239582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.239584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.239664 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.240048 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.240053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.240054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.240222 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.240328 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.240813 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.240948 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:02:17.241874 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.229023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45527 (local address 127.0.0.1:51982)
0620 15:02:17.229543 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.229582 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.230501 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.231068 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.231079 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.231428 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.232053 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.232068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.232891 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.232895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.234298 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.234317 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.234817 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.234821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.234992 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.235612 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.235627 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.237321 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.239081 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.239086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.239099 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.239448 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.239782 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.239785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.239787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.239923 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.240328 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.240335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.240572 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.240746 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.241041 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260620 15:02:17.243228 29767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab.
Entry for principal kudu/127.7.46.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab.
I20260620 15:02:17.247510 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.188:46241
--webserver_interface=127.7.46.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab
--principal=kudu/127.7.46.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:45527,127.7.46.189:41567,127.7.46.188:46241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
I20260620 15:02:17.252499 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.243760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55514)
0620 15:02:17.243870 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.243886 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.243958 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.244523 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.244526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.244685 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.244774 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.244779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.245868 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.245871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.246516 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.246532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.247202 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.247204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.247298 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.247726 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.247737 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.249079 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.250931 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.250937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.250939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.251118 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.251401 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.251406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.251408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.251504 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.251975 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.251978 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.252059 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.252194 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.252311 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":1}
I20260620 15:02:17.252517 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.243858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.7.46.189:41567)
0620 15:02:17.244151 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.244159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.244175 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:02:17.244258 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.244263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.244409 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.244540 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.244876 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.245755 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.246630 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.247082 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.247196 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.249213 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.249221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.249230 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.249248 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.250832 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.251232 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.251244 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.251245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.251312 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.251592 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.251598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.251600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.251877 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.252015 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.252265 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.252380 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260620 15:02:17.254101 29767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:17.254751 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.254522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55904)
0620 15:02:17.254629 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.254701 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:02:17.254947 29717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
I20260620 15:02:17.255710 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.255514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55916)
0620 15:02:17.255611 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.255654 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 15:02:17.255889 29767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:46241: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
I20260620 15:02:17.266858 29692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } attempt: 3
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967736, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
I20260620 15:02:17.275529 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.267327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.7.46.189:41567)
0620 15:02:17.267452 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.267455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.267467 (+ 12us) server_negotiation.cc:407] Connection header received
0620 15:02:17.267602 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.267604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.267639 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.267702 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.268169 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.268685 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.270177 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.270415 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.270508 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.272604 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.272607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.272609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.272628 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.273984 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.274400 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.274405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.274407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.274482 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.274762 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.274765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.274767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.274924 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.275034 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.275292 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.275393 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:02:17.275530 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.267276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55520)
0620 15:02:17.267407 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.267421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.267506 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.267745 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.267748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.267944 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.268059 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.268066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.268789 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.268792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.270065 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.270075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.270519 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.270522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.270633 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.271136 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.271151 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.272459 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.274104 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.274107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.274109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.274304 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.274581 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.274584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.274586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.274669 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.275026 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.275029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.275128 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.275232 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.275360 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:02:17.276612 29692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:17.277226 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.277013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55924)
0620 15:02:17.277120 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.277175 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:02:17.277949 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.277727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55934)
0620 15:02:17.277861 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.277902 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 15:02:17.278108 29692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:46241: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
I20260620 15:02:17.305114 29767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } attempt: 1
I20260620 15:02:17.305859 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.305568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55946)
0620 15:02:17.305714 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.305778 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:02:17.306607 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.306354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55958)
0620 15:02:17.306465 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.306520 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 15:02:17.306743 29767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:46241: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
I20260620 15:02:17.309260 29692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } attempt: 1
I20260620 15:02:17.309993 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.309682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55970)
0620 15:02:17.309887 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.309935 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 15:02:17.310621 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.310421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55974)
0620 15:02:17.310520 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.310571 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 15:02:17.310739 29692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:46241: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
W20260620 15:02:17.355708 29773 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:02:17.356019 29773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:17.356093 29773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:17.356173 29773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:17.360144 29773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:17.360220 29773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:17.360256 29773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:17.360314 29773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:17.360354 29773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:17.365316 29773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:45527,127.7.46.189:41567,127.7.46.188:46241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab
--principal=kudu/127.7.46.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.188:46241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.46.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:17.366258 29773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:17.367192 29773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:02:17.374914 29767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } attempt: 2
W20260620 15:02:17.375198 29779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:17.375181 29782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:17.375249 29780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:17.375459 29773 server_base.cc:1034] running on GCE node
I20260620 15:02:17.375834 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.375517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55982)
0620 15:02:17.375662 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.375712 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 15:02:17.376423 29773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:17.376551 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.376303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:55998)
0620 15:02:17.376404 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.376448 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 15:02:17.376705 29767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:46241: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
I20260620 15:02:17.377597 29773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:17.378769 29773 hybrid_clock.cc:648] HybridClock initialized: now 1781967737378755 us; error 34 us; skew 500 ppm
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:17.381299 29773 init.cc:377] Logged in from keytab as kudu/127.7.46.188@KRBTEST.COM (short username kudu)
I20260620 15:02:17.382084 29773 webserver.cc:460] Webserver started at http://127.7.46.188:43261/ using document root <none> and password file <none>
I20260620 15:02:17.382598 29773 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:17.382656 29773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:17.382879 29773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:17.384819 29773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/data/instance:
uuid: "2ab9d58207e24f54817e39b4da67e18c"
format_stamp: "Formatted at 2026-06-20 15:02:17 on dist-test-slave-30t4"
server_key: "6ed54152a7f3550d920590b5a0a98497"
server_key_iv: "73e1de872f058c98bf8bb96905d5f3a4"
server_key_version: "encryptionkey@0"
I20260620 15:02:17.385357 29773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/wal/instance:
uuid: "2ab9d58207e24f54817e39b4da67e18c"
format_stamp: "Formatted at 2026-06-20 15:02:17 on dist-test-slave-30t4"
server_key: "6ed54152a7f3550d920590b5a0a98497"
server_key_iv: "73e1de872f058c98bf8bb96905d5f3a4"
server_key_version: "encryptionkey@0"
I20260620 15:02:17.385875 29692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } attempt: 2
I20260620 15:02:17.386482 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.386248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:56006)
0620 15:02:17.386363 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.386419 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 15:02:17.387120 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.386922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:56008)
0620 15:02:17.387027 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.387076 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 15:02:17.387260 29692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:46241: Network error: Client connection negotiation failed: client connection to 127.7.46.188:46241: connect: Connection refused (error 111)
I20260620 15:02:17.389010 29773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:02:17.391192 29789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:17.392365 29773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:02:17.392495 29773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/wal
uuid: "2ab9d58207e24f54817e39b4da67e18c"
format_stamp: "Formatted at 2026-06-20 15:02:17 on dist-test-slave-30t4"
server_key: "6ed54152a7f3550d920590b5a0a98497"
server_key_iv: "73e1de872f058c98bf8bb96905d5f3a4"
server_key_version: "encryptionkey@0"
I20260620 15:02:17.392601 29773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:17.411530 29773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:17.414366 29773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:17.414561 29773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:17.421379 29773 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.188:46241
I20260620 15:02:17.421661 29841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.188:46241 every 8 connection(s)
I20260620 15:02:17.422358 29773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
I20260620 15:02:17.423028 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 29773
I20260620 15:02:17.423169 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967464421185-7354-0/minicluster-data/master-2/wal/instance
I20260620 15:02:17.423456 7354 external_mini_cluster.cc:1428] Setting key 44ff6b788dd97f27b82fba9f8a83aebd
I20260620 15:02:17.425446 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:17.428946 29842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:02:17.437567 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.424831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.7.46.188:46241)
0620 15:02:17.425233 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.425246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.425280 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:02:17.425965 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.425983 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.426286 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.426612 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.427395 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.428425 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.429320 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.429764 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.429943 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.432747 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.432754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.432769 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.432794 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.434560 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.434984 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.434989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.434995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.435104 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.435383 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.435386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.435388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.435816 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.436004 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.436277 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.436565 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260620 15:02:17.439262 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.430566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.7.46.190:45527)
0620 15:02:17.430718 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.430723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.431015 (+ 292us) server_negotiation.cc:407] Connection header received
0620 15:02:17.431155 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.431159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.431220 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.431310 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.432086 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.432647 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.433543 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.433782 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.433893 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.435906 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.435910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.435912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.435938 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.437570 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.438069 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.438072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.438074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.438160 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.438410 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.438414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.438417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.438639 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.438717 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.439046 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.439132 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:02:17.439386 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.430498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:45527 (local address 127.0.0.1:51994)
0620 15:02:17.430883 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.430928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.431049 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.431356 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.431360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.431763 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.431951 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.431959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.432776 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.432780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.433419 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.433427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.433894 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.433897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.434016 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.434480 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.434492 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.435729 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.437694 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.437706 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.437712 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.437973 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.438249 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.438251 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.438252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.438324 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.438739 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.438744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.438859 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.438998 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.439162 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":147,"threads_started":1}
I20260620 15:02:17.440641 29842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:02:17.448534 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.441074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:41567 (local address 127.0.0.1:55534)
0620 15:02:17.441177 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.441193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.441260 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.441615 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.441617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.441736 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.441818 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.441822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.442525 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.442528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.443005 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.443012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.443463 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.443466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.443578 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.444104 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.444115 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.445438 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.447083 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.447089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.447094 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.447328 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.447590 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.447592 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.447593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.447654 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.448068 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.448071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.448172 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.448289 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.448409 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 15:02:17.448652 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.441267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.7.46.189:41567)
0620 15:02:17.441410 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.441414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.441428 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:02:17.441468 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.441471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.441513 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.441592 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.441931 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.442422 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.443124 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.443356 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.443464 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.445570 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.445573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.445575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.445597 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.446980 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.447423 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.447427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.447428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.447511 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.447733 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.447737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.447739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.447969 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.448070 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.448333 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.448468 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:02:17.449363 29842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:17.450063 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.440122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.7.46.190:45527)
0620 15:02:17.440233 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.440237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.440266 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:02:17.440365 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.440368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.440406 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.440478 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.441194 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.441825 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.443163 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.443373 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.443480 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.446262 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.446265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.446268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.446297 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.447944 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.448409 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.448412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.448415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.448495 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.448871 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.448874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.448876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.449111 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.449348 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.449462 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.449568 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
I20260620 15:02:17.458057 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.449847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:56030)
0620 15:02:17.449934 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.449947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.450011 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.450303 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.450305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.450440 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.450494 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.450498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.451738 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.451744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.452275 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.452283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.452739 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.452742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.452811 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.453250 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.453263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.454614 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.456518 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.456521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.456522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.456781 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.457078 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.457080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.457081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.457154 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.457561 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.457563 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.457680 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.457797 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.457927 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 15:02:17.458221 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.449897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.7.46.188:46241)
0620 15:02:17.450009 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.450012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.450028 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:02:17.450078 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.450080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.450110 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.450210 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.450598 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.451644 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.452385 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.452637 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.452740 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.454738 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.454741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.454744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.454765 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.456396 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.456881 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.456886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.456888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.456973 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.457243 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.457248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.457252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.457462 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.457564 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.457848 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.458012 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":38}
I20260620 15:02:17.459225 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.451085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.7.46.189:41567)
0620 15:02:17.451186 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.451190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.451203 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:02:17.451255 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.451258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.451293 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.451372 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.451914 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.452384 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.453172 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.453519 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.453630 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.455863 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.455868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.455871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.455898 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.457548 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.457978 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.457983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.457985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.458071 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.458394 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.458397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.458399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.458597 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.458821 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.458966 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.459063 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 15:02:17.461489 29842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c: Bootstrap starting.
I20260620 15:02:17.462610 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:17.464170 29842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:17.464871 29842 log.cc:824] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:17.463965 29666 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "003c166f8822498aa53ff631ca29af61"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52002
I20260620 15:02:17.466933 29842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c: No bootstrap required, opened a new log
I20260620 15:02:17.467831 29767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } attempt: 3
I20260620 15:02:17.470454 29842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } }
I20260620 15:02:17.470638 29842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab9d58207e24f54817e39b4da67e18c, State: Initialized, Role: FOLLOWER
I20260620 15:02:17.470976 29842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } }
I20260620 15:02:17.471494 29848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.471698 29848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:17.472327 29842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
I20260620 15:02:17.477380 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.468236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:56042)
0620 15:02:17.468340 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.468356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.468434 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.468690 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.468693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.468885 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.468983 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.468989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.469954 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.469960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.470879 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.470888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.471446 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.471451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.471553 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.472051 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.472066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.473637 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.475707 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.475711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.475713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.475999 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.476326 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.476329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.476331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.476426 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.476849 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.476852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.476964 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.477075 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.477207 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 15:02:17.477563 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.468288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.7.46.188:46241)
0620 15:02:17.468396 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.468399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.468415 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:02:17.468522 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.468525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.468565 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.468656 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.469108 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.469841 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.470998 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.471330 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.471421 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.473784 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.473788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.473791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.473820 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.475566 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.476115 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.476119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.476121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.476221 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.476541 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.476548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.476549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.476758 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.476848 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.477168 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.477319 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 15:02:17.477927 29859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:17.477998 29859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:17.480444 29767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299: Bootstrap starting.
I20260620 15:02:17.476378 29773 master_runner.cc:186] Error getting master registration for 127.7.46.190:45527: OK, instance_id { permanent_uuid: "003c166f8822498aa53ff631ca29af61" instance_seqno: 1781967737025133 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:02:17.480785 29773 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:02:17.482982 29767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:17.483691 29767 log.cc:824] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:17.485390 29767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299: No bootstrap required, opened a new log
I20260620 15:02:17.486363 29692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } attempt: 3
I20260620 15:02:17.488891 29767 raft_consensus.cc:348] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } }
I20260620 15:02:17.489108 29767 raft_consensus.cc:729] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e889215feed14e6cb390ae8d84e5a299, State: Initialized, Role: FOLLOWER
I20260620 15:02:17.489413 29767 consensus_queue.cc:260] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } }
I20260620 15:02:17.489856 29862 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.489981 29862 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:17.491114 29767 sys_catalog.cc:564] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 15:02:17 dist-test-slave-30t4 krb5kdc[29611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967736, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
I20260620 15:02:17.495868 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.486771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:46241 (local address 127.0.0.1:56050)
0620 15:02:17.486891 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:17.486908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:17.486993 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:17.487260 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:17.487264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:17.487476 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:17.487608 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.487617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.488641 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.488645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.489424 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:17.489437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.490049 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.490053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:17.490135 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.490622 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:17.490637 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:17.492222 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:17.494162 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.494165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.494167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.494427 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.494733 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:17.494736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:17.494738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:17.494831 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:17.495276 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:17.495279 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:17.495394 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 15:02:17.495508 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 15:02:17.495666 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 15:02:17.496712 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.486853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.7.46.188:46241)
0620 15:02:17.486949 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.486954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.486969 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:02:17.487104 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.487108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.487151 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.487235 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.487735 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.488493 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.489564 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.489932 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.490057 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.492366 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.492370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.492373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.492400 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.494058 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.494523 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.494540 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.494542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.494632 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.494967 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.494974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.494976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.495176 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.495270 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.496424 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.496557 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 15:02:17.497738 29873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:17.497809 29873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:17.494886 29699 master_runner.cc:186] Error getting master registration for 127.7.46.190:45527: OK, instance_id { permanent_uuid: "003c166f8822498aa53ff631ca29af61" instance_seqno: 1781967737025133 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:02:17.498483 29699 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:02:17.501626 29692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61: Bootstrap starting.
I20260620 15:02:17.504334 29692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:17.505239 29692 log.cc:824] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:17.508072 29692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61: No bootstrap required, opened a new log
I20260620 15:02:17.513080 29692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } }
I20260620 15:02:17.513418 29692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003c166f8822498aa53ff631ca29af61, State: Initialized, Role: FOLLOWER
I20260620 15:02:17.513952 29692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } }
I20260620 15:02:17.514611 29876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.514804 29876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:17.516189 29666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:02:17.516324 29666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:17.517611 29666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } }
I20260620 15:02:17.516114 29692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:17.518568 29666 leader_election.cc:288] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [CANDIDATE]: Term 1 election: Requested vote from peers e889215feed14e6cb390ae8d84e5a299 (127.7.46.189:41567), 2ab9d58207e24f54817e39b4da67e18c (127.7.46.188:46241)
I20260620 15:02:17.519886 29741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "003c166f8822498aa53ff631ca29af61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e889215feed14e6cb390ae8d84e5a299"
I20260620 15:02:17.520159 29741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:17.521549 29741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 003c166f8822498aa53ff631ca29af61 in term 1.
W20260620 15:02:17.521716 29645 tablet.cc:2369] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:02:17.522478 29640 leader_election.cc:302] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003c166f8822498aa53ff631ca29af61, e889215feed14e6cb390ae8d84e5a299; no voters:
I20260620 15:02:17.522756 29876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:17.522982 29876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [term 1 LEADER]: Becoming Leader. State: Replica: 003c166f8822498aa53ff631ca29af61, State: Running, Role: LEADER
I20260620 15:02:17.523315 29876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } }
I20260620 15:02:17.523603 29816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "003c166f8822498aa53ff631ca29af61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2ab9d58207e24f54817e39b4da67e18c"
I20260620 15:02:17.523865 29816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:17.524987 29876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 003c166f8822498aa53ff631ca29af61. Latest consensus state: current_term: 1 leader_uuid: "003c166f8822498aa53ff631ca29af61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.525099 29876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:17.525193 29816 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 003c166f8822498aa53ff631ca29af61 in term 1.
I20260620 15:02:17.528764 29624 master_runner.cc:263] No leader master found from master 003c166f8822498aa53ff631ca29af61
I20260620 15:02:17.528832 29624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:02:17.533556 29894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:17.537271 29894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:17.537388 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.526125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.7.46.188:46241)
0620 15:02:17.526239 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.526243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.527982 (+ 1739us) server_negotiation.cc:407] Connection header received
0620 15:02:17.528177 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.528181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.528228 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.528281 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.529347 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.530127 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.531373 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.531707 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.531803 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.533642 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.533646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.533649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.533675 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.535398 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.535892 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.535896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.535898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.535981 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.536422 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.536426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.536428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.536640 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.536771 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.537119 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.537235 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 15:02:17.539469 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.528932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.7.46.190:45527)
0620 15:02:17.529032 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.529036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.529510 (+ 474us) server_negotiation.cc:407] Connection header received
0620 15:02:17.529697 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.529701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.529745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.529799 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.530788 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.531638 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.532852 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.533192 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.533277 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.535627 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.535631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.535634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.535660 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.537489 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.538044 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.538047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.538049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.538136 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.538457 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.538460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.538462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.538683 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.538797 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.539167 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.539279 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:02:17.547441 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:17.527278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.7.46.189:41567)
0620 15:02:17.527386 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.527390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.538784 (+ 11394us) server_negotiation.cc:407] Connection header received
0620 15:02:17.538940 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.538943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.538988 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.539051 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.539995 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.540760 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.541841 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:17.542183 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:17.542253 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:17.543974 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:17.543980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:17.543984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:17.544027 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:17.545702 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.546172 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.546175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.546177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.546255 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:17.546538 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:17.546541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:17.546543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:17.546735 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:17.546837 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:17.547185 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 15:02:17.547292 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:02:17.550750 29816 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [term 1 FOLLOWER]: Refusing update from remote peer 003c166f8822498aa53ff631ca29af61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:02:17.551035 29741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [term 1 FOLLOWER]: Refusing update from remote peer 003c166f8822498aa53ff631ca29af61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:02:17.551425 29892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:02:17.551633 29876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:02:17.555253 29848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 003c166f8822498aa53ff631ca29af61. Latest consensus state: current_term: 1 leader_uuid: "003c166f8822498aa53ff631ca29af61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.555383 29848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:17.555558 29862 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 003c166f8822498aa53ff631ca29af61. Latest consensus state: current_term: 1 leader_uuid: "003c166f8822498aa53ff631ca29af61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.555675 29862 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:17.562435 29876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "003c166f8822498aa53ff631ca29af61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.562577 29876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:17.564394 29862 sys_catalog.cc:455] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "003c166f8822498aa53ff631ca29af61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.564508 29862 sys_catalog.cc:458] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:17.564877 29899 mvcc.cc:205] Tried to move back new op lower bound from 7298939853001654272 to 7298939852898840576. Current Snapshot: MvccSnapshot[applied={T|T < 7298939853001654272}]
I20260620 15:02:17.567063 29892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "003c166f8822498aa53ff631ca29af61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.567186 29892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:17.567868 29901 mvcc.cc:205] Tried to move back new op lower bound from 7298939853001654272 to 7298939852898840576. Current Snapshot: MvccSnapshot[applied={T|T < 7298939853001654272}]
I20260620 15:02:17.568164 29848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "003c166f8822498aa53ff631ca29af61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003c166f8822498aa53ff631ca29af61" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 45527 } } peers { permanent_uuid: "e889215feed14e6cb390ae8d84e5a299" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 41567 } } peers { permanent_uuid: "2ab9d58207e24f54817e39b4da67e18c" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 46241 } } }
I20260620 15:02:17.568274 29848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:17.568717 29900 mvcc.cc:205] Tried to move back new op lower bound from 7298939853001654272 to 7298939852898840576. Current Snapshot: MvccSnapshot[applied={T|T < 7298939853001654272}]
I20260620 15:02:17.573890 29894 catalog_manager.cc:1347] Generated new cluster ID: 550d34373598453d8f7068f5f82df3d7
I20260620 15:02:17.573961 29894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:17.600607 29894 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:17.602026 29894 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:17.622409 29894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 003c166f8822498aa53ff631ca29af61: Generated new TSK 0
I20260620 15:02:17.623413 29894 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:02:17.661007 29845 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.657908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.7.46.188:46241)
0620 15:02:17.658047 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.658052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.659937 (+ 1885us) server_negotiation.cc:407] Connection header received
0620 15:02:17.660141 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.660144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.660192 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.660245 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.660871 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56074: BlockingRecv error: recv got EOF from 127.0.0.1:56074 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 15:02:17.662330 29696 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.658677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.7.46.190:45527)
0620 15:02:17.658785 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.658789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.661478 (+ 2689us) server_negotiation.cc:407] Connection header received
0620 15:02:17.661644 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.661647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.661694 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.661743 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.662217 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52026: BlockingRecv error: recv got EOF from 127.0.0.1:52026 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 15:02:17.663653 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.659354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.7.46.189:41567)
0620 15:02:17.659463 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.659468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.662759 (+ 3291us) server_negotiation.cc:407] Connection header received
0620 15:02:17.662983 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.662986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.663047 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.663096 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.663540 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55564: BlockingRecv error: recv got EOF from 127.0.0.1:55564 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 15:02:17.667847 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.666463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.7.46.189:41567)
0620 15:02:17.666567 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.666571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.666958 (+ 387us) server_negotiation.cc:407] Connection header received
0620 15:02:17.667169 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.667173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.667242 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.667291 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.667740 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55576: BlockingRecv error: recv got EOF from 127.0.0.1:55576 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 15:02:17.672668 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.671233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.7.46.189:41567)
0620 15:02:17.671320 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.671324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.671740 (+ 416us) server_negotiation.cc:407] Connection header received
0620 15:02:17.671990 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.671993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.672047 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.672093 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.672568 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55580: BlockingRecv error: recv got EOF from 127.0.0.1:55580 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 15:02:17.677974 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.676630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.7.46.189:41567)
0620 15:02:17.676716 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.676720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.677136 (+ 416us) server_negotiation.cc:407] Connection header received
0620 15:02:17.677308 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.677311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.677355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.677402 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.677857 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55596: BlockingRecv error: recv got EOF from 127.0.0.1:55596 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 15:02:17.685178 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.683894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.7.46.189:41567)
0620 15:02:17.684008 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.684020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.684419 (+ 399us) server_negotiation.cc:407] Connection header received
0620 15:02:17.684580 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.684583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.684622 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.684666 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.685095 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55604: BlockingRecv error: recv got EOF from 127.0.0.1:55604 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 15:02:17.697109 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.695209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.7.46.189:41567)
0620 15:02:17.695336 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.695340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.695925 (+ 585us) server_negotiation.cc:407] Connection header received
0620 15:02:17.696130 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.696133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.696181 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.696249 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.697006 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55608: BlockingRecv error: recv got EOF from 127.0.0.1:55608 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 15:02:17.716674 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.715513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.7.46.189:41567)
0620 15:02:17.715658 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.715663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.715817 (+ 154us) server_negotiation.cc:407] Connection header received
0620 15:02:17.715986 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.715990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.716055 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.716156 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.716562 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55622: BlockingRecv error: recv got EOF from 127.0.0.1:55622 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 15:02:17.751864 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.750641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.7.46.189:41567)
0620 15:02:17.750788 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.750793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.750927 (+ 134us) server_negotiation.cc:407] Connection header received
0620 15:02:17.751104 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.751113 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.751161 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.751250 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.751694 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55634: BlockingRecv error: recv got EOF from 127.0.0.1:55634 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 15:02:17.819188 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.817860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.7.46.189:41567)
0620 15:02:17.817989 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.817994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.818171 (+ 177us) server_negotiation.cc:407] Connection header received
0620 15:02:17.818345 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.818349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.818426 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.818520 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.819065 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55642: BlockingRecv error: recv got EOF from 127.0.0.1:55642 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 15:02:17.950631 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:17.949390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.7.46.189:41567)
0620 15:02:17.949558 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:17.949563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:17.949691 (+ 128us) server_negotiation.cc:407] Connection header received
0620 15:02:17.949873 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:17.949876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:17.949925 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:17.950019 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:17.950513 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55652: BlockingRecv error: recv got EOF from 127.0.0.1:55652 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 15:02:18.209946 29772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:18.208708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.7.46.189:41567)
0620 15:02:18.208897 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:18.208902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:18.209095 (+ 193us) server_negotiation.cc:407] Connection header received
0620 15:02:18.209232 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:18.209234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:18.209283 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:18.209355 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:18.209816 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55668: BlockingRecv error: recv got EOF from 127.0.0.1:55668 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 15:02:18.479286 29859 catalog_manager.cc:1259] Loaded cluster ID: 550d34373598453d8f7068f5f82df3d7
I20260620 15:02:18.479373 29859 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c: loading cluster ID for follower catalog manager: success
I20260620 15:02:18.481035 29859 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c: acquiring CA information for follower catalog manager: success
I20260620 15:02:18.482146 29859 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ab9d58207e24f54817e39b4da67e18c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:02:18.499017 29873 catalog_manager.cc:1259] Loaded cluster ID: 550d34373598453d8f7068f5f82df3d7
I20260620 15:02:18.499220 29873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299: loading cluster ID for follower catalog manager: success
I20260620 15:02:18.501066 29873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299: acquiring CA information for follower catalog manager: success
I20260620 15:02:18.502373 29873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e889215feed14e6cb390ae8d84e5a299: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:02:18.728050 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:18.724142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.7.46.189:41567)
0620 15:02:18.724393 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:18.724399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:18.724478 (+ 79us) server_negotiation.cc:407] Connection header received
0620 15:02:18.724642 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:18.724645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:18.724774 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:18.724893 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:18.725157 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.725684 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.726535 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.726764 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.726876 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:18.727199 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:18.727633 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:18.727773 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:18.727838 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 15:02:18.727892 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260620 15:02:18.734452 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:18.730909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.7.46.188:46241)
0620 15:02:18.731116 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:18.731119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:18.731131 (+ 12us) server_negotiation.cc:407] Connection header received
0620 15:02:18.731249 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:18.731251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:18.731353 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:18.731449 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:18.731647 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.732209 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.733072 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.733291 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.733392 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:18.733690 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:18.734066 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:18.734191 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:18.734250 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 15:02:18.734301 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260620 15:02:18.741737 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:18.737233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.7.46.189:41567)
0620 15:02:18.737416 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:18.737422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:18.737591 (+ 169us) server_negotiation.cc:407] Connection header received
0620 15:02:18.737792 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:18.737795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:18.737842 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:18.737931 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:18.738248 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.739039 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.740565 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.740816 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.740892 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:18.741184 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:18.741266 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:18.741333 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:18.741517 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 15:02:18.741581 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 15:02:18.742355 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:18.737299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.7.46.190:45527)
0620 15:02:18.737650 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:18.737654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:18.737667 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:02:18.737756 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:18.737759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:18.737908 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:18.738022 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:18.739239 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.739750 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.740646 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.740901 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.741001 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:18.741287 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:18.741653 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:18.741764 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:18.741998 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 15:02:18.742077 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":67,"threads_started":1}
I20260620 15:02:18.742391 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:18.737287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.7.46.188:46241)
0620 15:02:18.737508 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:18.737511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:18.737877 (+ 366us) server_negotiation.cc:407] Connection header received
0620 15:02:18.738010 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:18.738016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:18.738062 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:18.738130 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:18.738380 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.739109 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.740946 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:18.741289 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:18.741394 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:18.741756 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:18.741833 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:18.741884 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:18.742210 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 15:02:18.742286 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260620 15:02:18.750553 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 29624
I20260620 15:02:18.758070 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 29699
I20260620 15:02:18.763993 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 29773
2026-06-20T15:02:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3061 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:18 dist-test-slave-30t4 krb5kdc[29988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:18 dist-test-slave-30t4 krb5kdc[29988](info): set up 2 sockets
Jun 20 15:02:18 dist-test-slave-30t4 krb5kdc[29988](info): commencing operation
krb5kdc: starting...
W20260620 15:02:19.792418 7354 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/test-user.keytab.
Jun 20 15:02:19 dist-test-slave-30t4 krb5kdc[29988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.46.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
Entry for principal kudu/127.7.46.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab.
I20260620 15:02:19.908340 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34187
--webserver_interface=127.7.46.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:34187,127.7.46.189:37877,127.7.46.188:36347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:20.015103 30001 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:02:20.015359 30001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:20.015381 30001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:20.015462 30001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:20.019393 30001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:20.019455 30001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:20.019474 30001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:20.019490 30001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:20.019505 30001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:20.024683 30001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:34187,127.7.46.189:37877,127.7.46.188:36347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.190.keytab
--principal=kudu/127.7.46.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.190:34187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.46.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:20.025682 30001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:20.026670 30001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:20.034986 30007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:20.035010 30001 server_base.cc:1034] running on GCE node
W20260620 15:02:20.035018 30006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:20.034963 30009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:20.035655 30001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:20.036691 30001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:20.037850 30001 hybrid_clock.cc:648] HybridClock initialized: now 1781967740037822 us; error 44 us; skew 500 ppm
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:20.040555 30001 init.cc:377] Logged in from keytab as kudu/127.7.46.190@KRBTEST.COM (short username kudu)
I20260620 15:02:20.041430 30001 webserver.cc:460] Webserver started at http://127.7.46.190:34143/ using document root <none> and password file <none>
I20260620 15:02:20.041957 30001 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:20.042017 30001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:20.042173 30001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:20.044193 30001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/data/instance:
uuid: "7d4ecf17e5e548759b7d1448b86d6c73"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "c4669264aea15627dfcf815cf867ffee"
server_key_iv: "f9a30e207e4a91d52e18efa49c24f6ca"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.044695 30001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/wal/instance:
uuid: "7d4ecf17e5e548759b7d1448b86d6c73"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "c4669264aea15627dfcf815cf867ffee"
server_key_iv: "f9a30e207e4a91d52e18efa49c24f6ca"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.048223 30001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:02:20.050423 30016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:20.051605 30001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:02:20.051729 30001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/wal
uuid: "7d4ecf17e5e548759b7d1448b86d6c73"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "c4669264aea15627dfcf815cf867ffee"
server_key_iv: "f9a30e207e4a91d52e18efa49c24f6ca"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.051888 30001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:20.084789 30001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:20.087961 30001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:20.088160 30001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:20.094664 30001 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.190:34187
I20260620 15:02:20.094936 30068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.190:34187 every 8 connection(s)
I20260620 15:02:20.095577 30001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:20.098240 30069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:20.101822 30069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:20.104338 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 30001
I20260620 15:02:20.104449 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-0/wal/instance
I20260620 15:02:20.104768 7354 external_mini_cluster.cc:1428] Setting key ee4cb84e848b7c0df5e5ab76d24dd5c4
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.46.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.189@KRBTEST.COM" created.
I20260620 15:02:20.116984 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.104655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.7.46.190:34187)
0620 15:02:20.104921 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.104933 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.104979 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:02:20.105963 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.105967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.106152 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.106295 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.107387 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.108288 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.109277 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.109968 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.110130 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.112528 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.112534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.112542 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.112563 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.114208 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.114629 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.114634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.114635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.114734 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.115060 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.115062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.115063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.115412 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.115514 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.115998 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.116210 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":154,"spinlock_wait_cycles":2432,"thread_start_us":88,"threads_started":1}
I20260620 15:02:20.116969 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.103905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34187 (local address 127.0.0.1:43984)
0620 15:02:20.104398 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.104459 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.105664 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.106363 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.106371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.106867 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.107229 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.107249 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.108396 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.108399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.109150 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.109159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.110121 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.110124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.110202 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.110775 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.110791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.112367 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.114307 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.114314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.114322 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.114536 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.114857 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.114859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.114860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.114966 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.115526 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.115531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.115639 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.115888 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.116131 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260620 15:02:20.119251 30069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:20.119997 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.119693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36728)
0620 15:02:20.119873 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.119945 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 15:02:20.120260 30017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
I20260620 15:02:20.121119 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.120922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36730)
0620 15:02:20.121023 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.121064 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:02:20.121341 30069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:37877: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab.
Entry for principal kudu/127.7.46.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab.
I20260620 15:02:20.130769 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.189:37877
--webserver_interface=127.7.46.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab
--principal=kudu/127.7.46.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:34187,127.7.46.189:37877,127.7.46.188:36347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:02:20.170583 30069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } attempt: 1
I20260620 15:02:20.171473 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.171179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36734)
0620 15:02:20.171337 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.171397 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:02:20.172240 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.172018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36748)
0620 15:02:20.172138 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.172188 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 15:02:20.172374 30069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:37877: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
W20260620 15:02:20.237178 30076 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:02:20.237464 30076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:20.237540 30076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:20.237612 30076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:02:20.240535 30069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } attempt: 2
I20260620 15:02:20.241302 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.241029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36760)
0620 15:02:20.241164 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.241232 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:02:20.241698 30076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:20.241765 30076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:20.241827 30076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:20.241883 30076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:20.241946 30076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:20.242105 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.241852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36772)
0620 15:02:20.241965 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.242037 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 15:02:20.242226 30069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.189:37877: Network error: Client connection negotiation failed: client connection to 127.7.46.189:37877: connect: Connection refused (error 111)
I20260620 15:02:20.247220 30076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:34187,127.7.46.189:37877,127.7.46.188:36347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.189.keytab
--principal=kudu/127.7.46.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.189:37877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.46.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:20.248272 30076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:20.249267 30076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:20.257494 30082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:20.257498 30081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:20.257668 30076 server_base.cc:1034] running on GCE node
W20260620 15:02:20.257498 30084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:20.258252 30076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:20.259251 30076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:20.260438 30076 hybrid_clock.cc:648] HybridClock initialized: now 1781967740260427 us; error 29 us; skew 500 ppm
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:20.263196 30076 init.cc:377] Logged in from keytab as kudu/127.7.46.189@KRBTEST.COM (short username kudu)
I20260620 15:02:20.264072 30076 webserver.cc:460] Webserver started at http://127.7.46.189:32789/ using document root <none> and password file <none>
I20260620 15:02:20.264600 30076 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:20.264688 30076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:20.264906 30076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:20.266809 30076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/data/instance:
uuid: "a858620000484f278ca228cf0290a3ab"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "1f22ddc15c4745bd79ac8d6820fd498d"
server_key_iv: "902da7b160bc90e04ee3ca7bd58d00b2"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.267354 30076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/wal/instance:
uuid: "a858620000484f278ca228cf0290a3ab"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "1f22ddc15c4745bd79ac8d6820fd498d"
server_key_iv: "902da7b160bc90e04ee3ca7bd58d00b2"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.270813 30076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 15:02:20.273093 30091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:20.274179 30076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:02:20.274319 30076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/wal
uuid: "a858620000484f278ca228cf0290a3ab"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "1f22ddc15c4745bd79ac8d6820fd498d"
server_key_iv: "902da7b160bc90e04ee3ca7bd58d00b2"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.274422 30076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:20.293248 30076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:20.297297 30076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:20.297555 30076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:20.306809 30076 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.189:37877
I20260620 15:02:20.307123 30143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.189:37877 every 8 connection(s)
I20260620 15:02:20.307917 30076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/data/info.pb
I20260620 15:02:20.311671 30144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:20.316164 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 30076
I20260620 15:02:20.316299 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-1/wal/instance
I20260620 15:02:20.316560 7354 external_mini_cluster.cc:1428] Setting key 3508f7eb766d6f975386a7420ad763a7
I20260620 15:02:20.317523 30144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.46.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.46.188@KRBTEST.COM" created.
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:02:20.333384 30069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } attempt: 3
I20260620 15:02:20.334069 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.320005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.7.46.190:34187)
0620 15:02:20.320160 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.320167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.321263 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 15:02:20.322735 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.322737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.322788 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.322851 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.324496 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.325382 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.326673 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.326932 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.327122 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.329784 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.329788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.329791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.329820 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.331648 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.332321 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.332324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.332325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.332400 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.332810 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.332812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.332813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.332973 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.333046 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.333783 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.333900 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:02:20.335088 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.320494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34187 (local address 127.0.0.1:43986)
0620 15:02:20.321085 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.321142 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.322490 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.323037 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.323045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.323572 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.324284 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.324306 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.325553 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.325556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.326540 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.326549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.327174 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.327179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.327356 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.328045 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.328066 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.329572 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.331809 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.331818 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.331834 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.332220 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.332499 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.332505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.332509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.332707 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.333081 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.333089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.333476 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.333690 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.334027 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":153,"threads_started":1}
I20260620 15:02:20.336627 30144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.46.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab.
Entry for principal kudu/127.7.46.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab.
I20260620 15:02:20.343930 7354 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLLAc44/build/debug/bin/kudu
/tmp/dist-test-taskLLAc44/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.188:36347
--webserver_interface=127.7.46.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab
--principal=kudu/127.7.46.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.46.148:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.46.190:34187,127.7.46.189:37877,127.7.46.188:36347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:02:20.345198 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.333837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36778)
0620 15:02:20.334131 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.334149 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.334229 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.335068 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.335071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.335240 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.335327 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.335333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.336647 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.336650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.337929 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.337937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.338720 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.338723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.338804 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.339263 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.339277 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.340841 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.343170 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.343176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.343181 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.343421 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.343819 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.343824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.343827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.343942 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.344577 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.344583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.344698 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.344812 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.344964 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260620 15:02:20.345319 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.334376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.7.46.189:37877)
0620 15:02:20.334729 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.334740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.334766 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:02:20.334799 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.334803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.334940 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.335088 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.335451 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.336516 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.338055 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.338611 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.338714 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.340947 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.340960 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.340975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.341002 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.343045 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.343561 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.343569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.343574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.343678 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.344057 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.344061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.344063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.344378 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.344527 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.344931 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.345119 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":259,"thread_start_us":179,"threads_started":1}
I20260620 15:02:20.346825 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.337099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36794)
0620 15:02:20.337200 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.337215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.337282 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.337778 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.337781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.337992 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.338184 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.338193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.339090 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.339093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.339981 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.339990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.340546 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.340550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.340693 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.341140 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.341155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.342532 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.345293 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.345296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.345298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.345543 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.345846 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.345851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.345853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.345943 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.346360 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.346366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.346454 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.346574 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.346689 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":49}
I20260620 15:02:20.347018 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.337254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.7.46.189:37877)
0620 15:02:20.337552 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.337560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.337573 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:02:20.337604 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.337606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.337642 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.337729 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.338309 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.338973 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.340142 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.340441 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.340547 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.342688 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.342692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.342697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.342724 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.345188 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.345660 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.345664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.345667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.345754 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.346037 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.346042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.346045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.346255 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.346368 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.346657 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.346777 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
I20260620 15:02:20.347653 30069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:20.347750 30144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:20.348362 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.348140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52640)
0620 15:02:20.348250 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.348298 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 15:02:20.349066 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.348839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52664)
0620 15:02:20.348964 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.349007 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 15:02:20.349258 30069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:36347: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
I20260620 15:02:20.352136 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.348212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52656)
0620 15:02:20.351923 (+ 3711us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.352032 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3614}
W20260620 15:02:20.352337 30095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
I20260620 15:02:20.353291 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.353027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52676)
0620 15:02:20.353156 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.353215 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:02:20.353477 30144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:36347: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
I20260620 15:02:20.380452 30069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } attempt: 1
I20260620 15:02:20.381295 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.380988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52680)
0620 15:02:20.381138 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.381213 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:02:20.382073 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.381826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52696)
0620 15:02:20.381948 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.382001 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:02:20.382239 30069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:36347: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
I20260620 15:02:20.402765 30144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } attempt: 1
I20260620 15:02:20.403650 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.403351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52702)
0620 15:02:20.403506 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.403568 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:02:20.404455 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.404250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52712)
0620 15:02:20.404349 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.404393 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 15:02:20.404663 30144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:36347: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
W20260620 15:02:20.453640 30151 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:02:20.453965 30151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:20.454030 30151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:20.454105 30151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:02:20.457399 30069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } attempt: 2
I20260620 15:02:20.458194 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.457903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52722)
0620 15:02:20.458050 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.458108 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:02:20.458598 30151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:20.458667 30151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:20.458734 30151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:20.458772 30151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:20.458809 30151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:20.458916 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.458706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52736)
0620 15:02:20.458794 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.458847 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 15:02:20.459039 30069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:36347: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
I20260620 15:02:20.464053 30151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.46.148:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.46.190:34187,127.7.46.189:37877,127.7.46.188:36347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/krb5kdc/kudu_127.7.46.188.keytab
--principal=kudu/127.7.46.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.46.188:36347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.46.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 15:02:20.464993 30151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:20.465926 30151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:02:20.472808 30144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } attempt: 2
I20260620 15:02:20.473660 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.473238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52744)
0620 15:02:20.473494 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.473564 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260620 15:02:20.474189 30157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:20.474318 30158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:20.474395 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.474156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52750)
0620 15:02:20.474265 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.474321 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 15:02:20.474505 30151 server_base.cc:1034] running on GCE node
W20260620 15:02:20.474584 30144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.46.188:36347: Network error: Client connection negotiation failed: client connection to 127.7.46.188:36347: connect: Connection refused (error 111)
W20260620 15:02:20.474638 30160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:20.475157 30151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:20.476207 30151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:20.477391 30151 hybrid_clock.cc:648] HybridClock initialized: now 1781967740477376 us; error 25 us; skew 500 ppm
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:20.480042 30151 init.cc:377] Logged in from keytab as kudu/127.7.46.188@KRBTEST.COM (short username kudu)
I20260620 15:02:20.480844 30151 webserver.cc:460] Webserver started at http://127.7.46.188:39981/ using document root <none> and password file <none>
I20260620 15:02:20.481318 30151 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:20.481376 30151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:20.481537 30151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:02:20.483379 30151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/data/instance:
uuid: "101341dfb00b48228ecb48a9420e012f"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "eef7d8939377adeb96390fcd160fa01a"
server_key_iv: "1c20c5a3a4dceb2296d85e79992f972b"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.483940 30151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/wal/instance:
uuid: "101341dfb00b48228ecb48a9420e012f"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "eef7d8939377adeb96390fcd160fa01a"
server_key_iv: "1c20c5a3a4dceb2296d85e79992f972b"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.487416 30151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:02:20.489884 30167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:20.491230 30151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:02:20.491328 30151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/data,/tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/wal
uuid: "101341dfb00b48228ecb48a9420e012f"
format_stamp: "Formatted at 2026-06-20 15:02:20 on dist-test-slave-30t4"
server_key: "eef7d8939377adeb96390fcd160fa01a"
server_key_iv: "1c20c5a3a4dceb2296d85e79992f972b"
server_key_version: "encryptionkey@0"
I20260620 15:02:20.491406 30151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:20.503299 30151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:20.506214 30151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:20.506361 30151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:20.513267 30151 rpc_server.cc:307] RPC server started. Bound to: 127.7.46.188:36347
I20260620 15:02:20.513557 30219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.46.188:36347 every 8 connection(s)
I20260620 15:02:20.514197 30151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/data/info.pb
I20260620 15:02:20.516930 30220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:20.520458 30220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:20.521179 7354 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLLAc44/build/debug/bin/kudu as pid 30151
I20260620 15:02:20.521307 7354 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLLAc44/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967464421185-7354-0/minicluster-data/master-2/wal/instance
I20260620 15:02:20.521597 7354 external_mini_cluster.cc:1428] Setting key c4ddf2b9b95d87c1bc1325e73c258a30
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
I20260620 15:02:20.534318 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.522359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.7.46.190:34187)
0620 15:02:20.522504 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.522509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.523236 (+ 727us) server_negotiation.cc:407] Connection header received
0620 15:02:20.524153 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.524156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.524211 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.524309 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.525370 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.526265 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.527584 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.527871 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.527964 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.530643 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.530647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.530653 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.530683 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.532335 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.532848 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.532854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.532856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.532937 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.533265 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.533269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.533271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.533461 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.533576 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.534033 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.534145 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:02:20.535113 30223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.522686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.190:34187 (local address 127.0.0.1:43988)
0620 15:02:20.523091 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.523132 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.523959 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.524359 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.524364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.524841 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.525199 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.525220 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.526405 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.526408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.527468 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.527478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.528009 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.528012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.528155 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.528729 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.528765 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.530490 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.532457 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.532465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.532474 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.532746 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.533036 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.533039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.533041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.533172 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.533573 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.533580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.533810 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.533977 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.534275 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":103,"threads_started":1}
I20260620 15:02:20.535118 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.523425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.7.46.188:36347)
0620 15:02:20.523670 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.523680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.523710 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:02:20.523970 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.523992 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.524179 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.524339 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.525324 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.526706 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.527799 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.528248 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.528326 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.531053 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.531058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.531068 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.531088 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.532640 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.533066 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.533079 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.533080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.533185 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.533472 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.533477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.533479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.533794 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.534015 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.534222 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.534359 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260620 15:02:20.536597 30220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.190@KRBTEST.COM
I20260620 15:02:20.545260 30223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.537120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.189:37877 (local address 127.0.0.1:36798)
0620 15:02:20.537230 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.537245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.537314 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.537706 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.537708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.537828 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.537915 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.537919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.538667 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.538671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.539163 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.539171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.539679 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.539682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.539813 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.540291 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.540307 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.541630 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.543519 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.543524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.543527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.543940 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.544246 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.544248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.544249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.544327 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.544724 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.544728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.544832 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.544928 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.545070 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:02:20.545289 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.537255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.7.46.189:37877)
0620 15:02:20.537384 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.537390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.537405 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:02:20.537450 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.537452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.537520 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.537596 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.538035 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.538551 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.539286 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.539561 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.539682 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.541772 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.541775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.541777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.541800 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.543413 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.544045 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.544049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.544051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.544141 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.544426 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.544430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.544434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.544629 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.544702 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.545011 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.545117 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 15:02:20.545869 30220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:02:20.546200 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.537564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.7.46.190:34187)
0620 15:02:20.537696 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.537700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.537713 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:02:20.537794 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.537797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.537841 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.537922 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.538471 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.538936 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.539672 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.539976 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.540079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.542690 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.542695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.542699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.542726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.544388 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.544814 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.544817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.544818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.544887 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.545202 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.545205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.545207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.545564 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.545791 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.545918 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.546024 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.188@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.46.189@KRBTEST.COM
I20260620 15:02:20.553853 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.546689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.7.46.188:36347)
0620 15:02:20.546829 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.546833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.546847 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:02:20.546881 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.546884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.546932 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.547033 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.547388 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.547939 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.548638 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.548865 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.548975 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.550732 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.550736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.550739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.550766 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.552318 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.552738 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.552744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.552746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.552823 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.553107 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.553112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.553113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.553269 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.553372 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.553606 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.553727 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":73}
I20260620 15:02:20.553851 30223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.546346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52762)
0620 15:02:20.546480 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.546493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.546563 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.547044 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.547046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.547196 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.547251 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.547255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.548036 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.548042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.548529 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.548536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.548971 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.548974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.549068 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.549522 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.549535 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.550616 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.552443 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.552447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.552449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.552643 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.552916 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.552919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.552921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.553015 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.553363 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.553366 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.553444 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.553552 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.553679 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":2}
I20260620 15:02:20.555949 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.547452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.7.46.189:37877)
0620 15:02:20.547556 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.547562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.547595 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:02:20.547726 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.547729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.547788 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.547854 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.548410 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.548873 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.549680 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.549943 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.550053 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.552817 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.552820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.552823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.552846 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.554373 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.554774 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.554779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.554780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.554857 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.555150 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.555154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.555156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.555316 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.555506 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.555669 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.555797 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 15:02:20.557976 30220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f: Bootstrap starting.
I20260620 15:02:20.558187 30069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } attempt: 3
I20260620 15:02:20.560271 30220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:20.560993 30220 log.cc:824] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:20.563158 7354 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:02:20.563537 30220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f: No bootstrap required, opened a new log
I20260620 15:02:20.564358 30043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7d4ecf17e5e548759b7d1448b86d6c73"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43996
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.190@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
I20260620 15:02:20.565828 30144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } attempt: 3
I20260620 15:02:20.567827 30220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } }
I20260620 15:02:20.568087 30220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101341dfb00b48228ecb48a9420e012f, State: Initialized, Role: FOLLOWER
I20260620 15:02:20.568471 30220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } }
I20260620 15:02:20.568948 30227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.569123 30227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:20.569818 30220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:20.569880 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.561535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52764)
0620 15:02:20.561692 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.561710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.561812 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.562046 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.562048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.562219 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.562315 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.562320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.563176 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.563179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.563869 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.563878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.564329 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.564333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.564413 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.564879 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.564893 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.566047 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.568224 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.568228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.568230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.568464 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.568759 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.568762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.568764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.568855 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.569403 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.569405 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.569477 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.569588 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.569715 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:02:20.569941 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.561621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.7.46.188:36347)
0620 15:02:20.561717 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.561722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.561758 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:02:20.561900 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.561903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.561945 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.562022 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.562424 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.563019 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.563979 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.564217 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.564327 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.566164 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.566168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.566171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.566196 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.568059 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.568568 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.568571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.568573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.568652 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.568964 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.568971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.568973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.569232 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.569309 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.569638 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.569739 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 15:02:20 dist-test-slave-30t4 krb5kdc[29988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967740, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.46.189@KRBTEST.COM for kudu/127.7.46.188@KRBTEST.COM
I20260620 15:02:20.573577 30069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73: Bootstrap starting.
I20260620 15:02:20.575533 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.566202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.46.188:36347 (local address 127.0.0.1:52772)
0620 15:02:20.566373 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:20.566387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:20.566466 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:20.566974 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:20.566977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:20.567158 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:20.567268 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.567278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.568321 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.568324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.569085 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:20.569096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.569794 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.569797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:20.569900 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.570400 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:20.570417 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:20.571890 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:20.574055 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.574058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.574060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.574294 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.574559 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:20.574563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:20.574566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:20.574653 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:20.575032 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:20.575046 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:20.575140 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 15:02:20.575238 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:02:20.575362 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 15:02:20.576252 30069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:20.575891 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.566248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.7.46.188:36347)
0620 15:02:20.566764 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.566776 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.566788 (+ 12us) server_negotiation.cc:407] Connection header received
0620 15:02:20.566825 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.566826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.566857 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.566959 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.567410 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.568221 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.569221 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.569560 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.569654 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.572033 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.572037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.572040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.572064 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.573942 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.574385 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.574389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.574390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.574471 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.574738 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.574741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.574743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.574940 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.575031 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.575300 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.575398 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":87,"threads_started":1}
W20260620 15:02:20.577369 30238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:20.577440 30238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:20.580046 30144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab: Bootstrap starting.
I20260620 15:02:20.577824 30151 master_runner.cc:186] Error getting master registration for 127.7.46.190:34187: OK, instance_id { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" instance_seqno: 1781967740092929 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:02:20.581238 30151 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:02:20.582417 30144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:20.583262 30144 log.cc:824] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:20.584524 30069 log.cc:824] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:20.585307 30144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab: No bootstrap required, opened a new log
I20260620 15:02:20.586575 30069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73: No bootstrap required, opened a new log
I20260620 15:02:20.588322 30144 raft_consensus.cc:348] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } }
I20260620 15:02:20.588523 30144 raft_consensus.cc:729] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a858620000484f278ca228cf0290a3ab, State: Initialized, Role: FOLLOWER
I20260620 15:02:20.588835 30144 consensus_queue.cc:260] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } }
I20260620 15:02:20.589341 30242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.589411 30069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } }
I20260620 15:02:20.589600 30069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4ecf17e5e548759b7d1448b86d6c73, State: Initialized, Role: FOLLOWER
I20260620 15:02:20.589722 30242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:20.589883 30069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } }
I20260620 15:02:20.590157 30144 sys_catalog.cc:564] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:20.590458 30243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.590772 30243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:20.591053 30069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:20.591140 30043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:02:20.591241 30043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:20.592576 30043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } }
I20260620 15:02:20.594080 30118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a858620000484f278ca228cf0290a3ab"
I20260620 15:02:20.594234 30194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "101341dfb00b48228ecb48a9420e012f"
I20260620 15:02:20.594291 30118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:20.594446 30194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:20.595358 30118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4ecf17e5e548759b7d1448b86d6c73 in term 1.
I20260620 15:02:20.595626 30194 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4ecf17e5e548759b7d1448b86d6c73 in term 1.
I20260620 15:02:20.593441 30043 leader_election.cc:288] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [CANDIDATE]: Term 1 election: Requested vote from peers a858620000484f278ca228cf0290a3ab (127.7.46.189:37877), 101341dfb00b48228ecb48a9420e012f (127.7.46.188:36347)
I20260620 15:02:20.595742 30017 leader_election.cc:302] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4ecf17e5e548759b7d1448b86d6c73, a858620000484f278ca228cf0290a3ab; no voters:
I20260620 15:02:20.599804 30243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:20.601763 30076 master_runner.cc:263] No leader master found from master a858620000484f278ca228cf0290a3ab
I20260620 15:02:20.601830 30076 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:02:20.601863 30001 master_runner.cc:263] No leader master found from master 7d4ecf17e5e548759b7d1448b86d6c73
I20260620 15:02:20.601928 30001 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 15:02:20.602708 30265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:20.602793 30265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:20.602928 30243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4ecf17e5e548759b7d1448b86d6c73, State: Running, Role: LEADER
I20260620 15:02:20.603211 30243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } }
I20260620 15:02:20.605608 30273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d4ecf17e5e548759b7d1448b86d6c73. Latest consensus state: current_term: 1 leader_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.605739 30273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: This master's current role is: LEADER
W20260620 15:02:20.607386 30262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:02:20.607462 30262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:02:20.614064 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.598981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.7.46.189:37877)
0620 15:02:20.599277 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.599281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.604809 (+ 5528us) server_negotiation.cc:407] Connection header received
0620 15:02:20.604961 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.604964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.605018 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.605085 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.606243 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.606979 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.608226 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.608549 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.608625 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.610453 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.610457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.610460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.610489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.612210 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.612698 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.612702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.612704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.612784 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.613104 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.613107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.613109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.613314 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.613426 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.613782 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.613892 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260620 15:02:20.617524 30276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:20.618436 30276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:20.625157 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.604103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.7.46.188:36347)
0620 15:02:20.604222 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.604226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.615742 (+ 11516us) server_negotiation.cc:407] Connection header received
0620 15:02:20.615935 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.615939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.615988 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.616053 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.616899 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.618131 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.619418 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.619802 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.619883 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.621554 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.621558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.621561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.621588 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.623298 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.623820 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.623823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.623825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.623912 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.624222 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.624225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.624227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.624422 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.624524 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.624868 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.624985 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 15:02:20.627681 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:20.598482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.7.46.190:34187)
0620 15:02:20.598587 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.598592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.604507 (+ 5915us) server_negotiation.cc:407] Connection header received
0620 15:02:20.604647 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.604650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.604695 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.604765 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.619863 (+ 15098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.620686 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.621840 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:20.622167 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:20.622236 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:20.623929 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:20.623933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:20.623935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:20.623960 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:20.625605 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.626083 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.626086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.626088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.626175 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:20.626482 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:20.626486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:20.626487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:20.626705 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:20.626817 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:20.627426 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 15:02:20.627534 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:02:20.635380 30193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [term 1 FOLLOWER]: Refusing update from remote peer 7d4ecf17e5e548759b7d1448b86d6c73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:02:20.636113 30273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:02:20.640268 30227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d4ecf17e5e548759b7d1448b86d6c73. Latest consensus state: current_term: 1 leader_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.640391 30227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:20.641404 30118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [term 1 FOLLOWER]: Refusing update from remote peer 7d4ecf17e5e548759b7d1448b86d6c73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:02:20.641966 30273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:02:20.645661 30242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d4ecf17e5e548759b7d1448b86d6c73. Latest consensus state: current_term: 1 leader_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.645771 30242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:20.649906 30243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.650027 30243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:20.650293 30278 mvcc.cc:205] Tried to move back new op lower bound from 7298939865634766848 to 7298939865513865216. Current Snapshot: MvccSnapshot[applied={T|T < 7298939865634766848}]
I20260620 15:02:20.650276 30242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.650373 30242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:20.652132 30281 mvcc.cc:205] Tried to move back new op lower bound from 7298939865634766848 to 7298939865513865216. Current Snapshot: MvccSnapshot[applied={T|T < 7298939865634766848}]
I20260620 15:02:20.652791 30273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.652901 30273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:20.653682 30227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4ecf17e5e548759b7d1448b86d6c73" member_type: VOTER last_known_addr { host: "127.7.46.190" port: 34187 } } peers { permanent_uuid: "a858620000484f278ca228cf0290a3ab" member_type: VOTER last_known_addr { host: "127.7.46.189" port: 37877 } } peers { permanent_uuid: "101341dfb00b48228ecb48a9420e012f" member_type: VOTER last_known_addr { host: "127.7.46.188" port: 36347 } } }
I20260620 15:02:20.653789 30227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:02:20.656888 30279 mvcc.cc:205] Tried to move back new op lower bound from 7298939865634766848 to 7298939865513865216. Current Snapshot: MvccSnapshot[applied={T|T < 7298939865634766848}]
I20260620 15:02:20.659394 30276 catalog_manager.cc:1347] Generated new cluster ID: a5b15440a10240b7a65c5739c0695720
I20260620 15:02:20.659456 30276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:20.682355 30276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:20.683561 30276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:20.695640 30276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d4ecf17e5e548759b7d1448b86d6c73: Generated new TSK 0
I20260620 15:02:20.696514 30276 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:02:20.733623 30226 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.731850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.7.46.188:36347)
0620 15:02:20.731971 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.731976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.732548 (+ 572us) server_negotiation.cc:407] Connection header received
0620 15:02:20.732753 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.732757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.732805 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.732856 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.733491 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52790: BlockingRecv error: recv got EOF from 127.0.0.1:52790 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 15:02:20.735880 30072 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.734291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.7.46.190:34187)
0620 15:02:20.734401 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.734406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.734959 (+ 553us) server_negotiation.cc:407] Connection header received
0620 15:02:20.735153 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.735156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.735205 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.735256 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.735747 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44020: BlockingRecv error: recv got EOF from 127.0.0.1:44020 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 15:02:20.737100 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.734750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.7.46.189:37877)
0620 15:02:20.734870 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.734874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.736264 (+ 1390us) server_negotiation.cc:407] Connection header received
0620 15:02:20.736382 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.736385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.736436 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.736514 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.736984 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36818: BlockingRecv error: recv got EOF from 127.0.0.1:36818 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 15:02:20.742005 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.740353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.7.46.189:37877)
0620 15:02:20.740464 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.740470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.741042 (+ 572us) server_negotiation.cc:407] Connection header received
0620 15:02:20.741264 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.741267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.741345 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.741399 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.741905 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36822: BlockingRecv error: recv got EOF from 127.0.0.1:36822 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 15:02:20.750705 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.747052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.7.46.189:37877)
0620 15:02:20.747183 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.747187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.748430 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 15:02:20.749046 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.749051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.749140 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.749236 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.750570 (+ 1334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36832: BlockingRecv error: recv got EOF from 127.0.0.1:36832 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 15:02:20.757306 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.755757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.7.46.189:37877)
0620 15:02:20.755889 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.755894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.756390 (+ 496us) server_negotiation.cc:407] Connection header received
0620 15:02:20.756604 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.756607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.756661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.756714 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.757206 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36842: BlockingRecv error: recv got EOF from 127.0.0.1:36842 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 15:02:20.765072 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.763513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.7.46.189:37877)
0620 15:02:20.763638 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.763642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.764140 (+ 498us) server_negotiation.cc:407] Connection header received
0620 15:02:20.764322 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.764327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.764382 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.764445 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.764976 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36858: BlockingRecv error: recv got EOF from 127.0.0.1:36858 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:02:20.776803 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.775241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.7.46.189:37877)
0620 15:02:20.775339 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.775344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.775868 (+ 524us) server_negotiation.cc:407] Connection header received
0620 15:02:20.776069 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.776072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.776124 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.776179 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.776710 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36870: BlockingRecv error: recv got EOF from 127.0.0.1:36870 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 15:02:20.796576 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.795046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36886 (local address 127.7.46.189:37877)
0620 15:02:20.795149 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.795153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.795602 (+ 449us) server_negotiation.cc:407] Connection header received
0620 15:02:20.795863 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.795866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.795915 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.795969 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.796457 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36886: BlockingRecv error: recv got EOF from 127.0.0.1:36886 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 15:02:20.832247 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.831014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.7.46.189:37877)
0620 15:02:20.831146 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.831151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.831378 (+ 227us) server_negotiation.cc:407] Connection header received
0620 15:02:20.831505 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.831508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.831561 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.831633 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.832124 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36902: BlockingRecv error: recv got EOF from 127.0.0.1:36902 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 15:02:20.899432 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:20.898219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.7.46.189:37877)
0620 15:02:20.898383 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:20.898388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:20.898527 (+ 139us) server_negotiation.cc:407] Connection header received
0620 15:02:20.898701 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:20.898704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:20.898763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:20.898854 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:20.899307 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36914: BlockingRecv error: recv got EOF from 127.0.0.1:36914 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 15:02:21.030625 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:21.029561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.7.46.189:37877)
0620 15:02:21.029741 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:21.029746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:21.029762 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:02:21.029910 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:21.029912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:21.029961 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:21.030043 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:21.030473 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36928: BlockingRecv error: recv got EOF from 127.0.0.1:36928 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 15:02:21.290014 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:02:21.288850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.7.46.189:37877)
0620 15:02:21.289018 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:21.289023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:21.289155 (+ 132us) server_negotiation.cc:407] Connection header received
0620 15:02:21.289281 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:21.289283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:21.289334 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:21.289412 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:21.289877 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36942: BlockingRecv error: recv got EOF from 127.0.0.1:36942 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:02:21.578739 30238 catalog_manager.cc:1259] Loaded cluster ID: a5b15440a10240b7a65c5739c0695720
I20260620 15:02:21.578814 30238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f: loading cluster ID for follower catalog manager: success
I20260620 15:02:21.580500 30238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f: acquiring CA information for follower catalog manager: success
I20260620 15:02:21.581622 30238 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 101341dfb00b48228ecb48a9420e012f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:02:21.608800 30262 catalog_manager.cc:1259] Loaded cluster ID: a5b15440a10240b7a65c5739c0695720
I20260620 15:02:21.608861 30262 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab: loading cluster ID for follower catalog manager: success
I20260620 15:02:21.610478 30262 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab: acquiring CA information for follower catalog manager: success
I20260620 15:02:21.611589 30262 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a858620000484f278ca228cf0290a3ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:02:21.808436 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:21.804045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.7.46.189:37877)
0620 15:02:21.804311 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:21.804316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:21.804437 (+ 121us) server_negotiation.cc:407] Connection header received
0620 15:02:21.804615 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:21.804619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:21.804755 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:21.804873 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:21.805159 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.805684 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.806726 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.807099 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.807193 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:21.807540 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:21.808042 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:21.808179 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:21.808220 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 15:02:21.808274 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260620 15:02:21.815255 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:21.811437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.7.46.188:36347)
0620 15:02:21.811701 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:21.811704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:21.811848 (+ 144us) server_negotiation.cc:407] Connection header received
0620 15:02:21.811996 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:21.811998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:21.812118 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:21.812215 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:21.812382 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.812865 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.813822 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.814047 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.814139 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:21.814416 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:21.814817 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:21.814911 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:21.814979 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 15:02:21.815074 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260620 15:02:21.822427 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:21.818350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.7.46.188:36347)
0620 15:02:21.818501 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:21.818505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:21.818820 (+ 315us) server_negotiation.cc:407] Connection header received
0620 15:02:21.819100 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:21.819104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:21.819155 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:21.819223 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:21.819527 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.820181 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.821248 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.821568 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.821661 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:21.821996 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:21.822064 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:21.822143 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:21.822232 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 15:02:21.822276 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:02:21.822543 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:21.818787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.7.46.189:37877)
0620 15:02:21.818911 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:21.818914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:21.819316 (+ 402us) server_negotiation.cc:407] Connection header received
0620 15:02:21.819418 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:21.819421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:21.819470 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:21.819548 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:21.819858 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.820509 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.821364 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.821694 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.821793 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:21.822059 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:21.822126 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:21.822171 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:21.822356 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 15:02:21.822417 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 15:02:21.822847 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:21.818269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.7.46.190:34187)
0620 15:02:21.818639 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:21.818644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:21.818660 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:02:21.818754 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:21.818757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:21.818905 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:21.819000 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:21.819247 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.820013 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.821189 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:21.821492 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:21.821598 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:21.821917 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:21.822228 (+ 311us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:21.822313 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:21.822644 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 15:02:21.822705 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":72,"threads_started":1}
I20260620 15:02:21.825939 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 30001
I20260620 15:02:21.832523 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 30076
I20260620 15:02:21.838378 7354 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLLAc44/build/debug/bin/kudu with pid 30151
2026-06-20T15:02:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3074 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6135 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277411 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST